Bertrand Schneiderhttp://blog.bertrandschneider.com
Thoughts on Education and TechnologyFri, 16 Mar 2018 19:30:11 +0000en-UShourly1https://wordpress.org/?v=4.9.4Studying in-situ Joint Visual Attentionhttp://blog.bertrandschneider.com/?p=761
http://blog.bertrandschneider.com/?p=761#commentsSat, 29 Nov 2014 22:12:37 +0000http://blog.bertrandschneider.com/?p=761The goal of this work is to describe a methodology for synchronizing two eye-tracking goggles and computing measures of joint visual attention (JVA) in a co-located setting. In our study, dyads of students interacted with different version of a tangible interface designed for students in logistics.

The video above shows the video recorded by each mobile eye-tracker and a ground truth on the bottom. We use common points known in each perspective (i.e., the two mobile eye-trackers and the top down view) to perform an homography and remap students’ gazes onto the ground truth. The points used for the homography are the four corners of each fiducial marker detected on top of the small scale shelves. On the right side, a cross recurrence graph shows moments of joint visual attention with the location of this moment: red indicates the first warehouse, green the second one, and blue the last warehouse. Cross-recurrence graphs show time for the first participant on the x-axis, time for the second participant on the y-axis, and colored pixels when the two participants are looking at the same location. Thus, a dark diagonal represents synchronized moments of joint attention, while off-diagonal pixels show moments of joint attention with a time lag.

Preliminary results are reported in Schneider et al. (accepted): we found that this measure of joint visual attention is a good proxy for the smoothness of the collaboration between two students, and that this measure is significantly correlated to participants’ performance (and in some cases, their learning gain).

One of my goals is to explore the affordances of new technologies for learning STEM disciplines (Science, Technology, Engineering and Mathematics). In collaboration with others, we are building a series of tangible interfaces for teaching science concepts. In this project, we started with the constraint of teaching a highly spatial domain where one could take advantage of the “3Dness” of physical objects. Two of the creators had been trained in neuroscience and suggested the brain as an ideal domain for designing educational TUIs. We interviewed novices after they had read a text explaining how the hearing system works and asked them various questions about how information is transformed at each step of the process. We found that novices had several misconceptions about the hearing system: first, they had trouble visualizing the different transductions happening in the ear (i.e. sound waves vibrate the ear drum with various sound pressures; the ear drum then moves the maleus to pass information as a mechanical movement; the ear bones then move the liquid contained in the cochlea and activate particular segments of the basilar membrane rolled in the cochlea). Secondly, novices struggled with the spatial mapping of different sound frequencies on the basilar membrane. High frequencies carry more energy and vibrate thicker segments of the membrane, while low frequency sounds traverse the membrane until it finds a segment thin enough to be activated. This mapping is counter-intuitive for novices, because we usually represent sounds on a number line from low (left) to high (right) frequency. On the basilar membrane, this mapping is reversed.

The design of our first prototype focused on those two aspects: the propagation and transduction of sounds through the hearing system, and the spatial mapping of sound frequencies on the basilar membrane.

Design of the System

This section describes the first prototype of EarExplorer. The system is shown below. Each tangible was created using a low-cost 3D printer. A projector displays an augmented reality layer by reflecting its image on a mirror held above the tabletop. A camera is attached to the mirror and detects the location of the fiducials on the tangibles.

When starting the activity, users are presented with three elements on the table: the outer ear, which is the starting point of the activity (top left corner, Fig. 2; the auditory cortex, which is the end point of the activity (bottom right corner, Fig. 2); and an information box (bottom left corner, Fig. 2). Eight tangibles are arranged around the projected area. Students are asked to connect the tangibles between the starting point and the ending point to let sound waves reach the auditory cortex. The 8 tangibles (in bold below) serve the following functions:

1) The speaker generates sound waves at four different frequencies (low, medium, high, very high). Those four frequencies are displayed on top of the speaker with a specific color coding (from low to high frequency: blue, green, yellow, red). By flipping the speaker, users can generate a series of sound waves to test their system.

2) The ear canal then needs to be linked to the starting point of the activity (the outer ear) and carries sound waves to the eardrum. There are two feedback showing that the tangibles are successfully connected: first, students see the sound waves follow the ear canal; second, they also see the eardrum move back and forth in the augmented reality view as the sound waves reach the end of the ear canal.

3) The ear bones need to be connected to the ear canal. As the eardrum vibrates back and forth, the ear bones will provide a similar feedback: the augmented reality view will project the shape of the ear bones on the tangible and animate them back and forth as the sound waves are reaching this part of the auditory circuit.

Figure 1: The EarExplorer Interface, after the users have connected all the tangibles in the correct sequence. They use the infobox (1) to learn about the different organs and connect them together; they then generate sounds at different frequencies with a speaker (2); sound waves travel from the emitter through the ear canal to the ear bones (3); finally, the sound reaches the basilar membrane inside the cochlea, activates a specific neuron and replays the sound if the configuration is correct (4).

4) The snail-shaped part of the cochlea contains the basilar membrane, which react to different sounds frequencies: the base is thicker and reacts to high-energy (high frequency) sounds; the apex (i.e. tail) is thinner and react to low-energy (low frequency) sounds. When students connect the cochlea to the ear bones, they see the basilar membrane being unrolled bellow the tangible. Since understanding this step is such a crucial moment, we display a small video of a teacher reiterating that the membrane is unrolled to facilitate their task; he also instructs the students to rebuild the basilar membrane by ordering the four tangible neurons.

5-8) In this step, four neurons need to be correctly sequenced bellow the cochlea to rebuild the basilar membrane. Each neuron is associated with a particular thickness of the membrane, and a particular sound frequency. Each neuron is color-coded according to the coding scheme displayed on top of the speaker (from low to high frequencies: blue, green, yellow, red). We simplified the behavior of the system to provide an intuitive feedback when a sound wave reaches the basilar membrane: if the order is correct, users will see the part of the membrane associated with this neuron vibrate, followed by electrical potentials travelling through the neuron (Fig.2, bottom right corner, blue neuron), and the audio sound being replayed.

Each tangible can be positioned in the information box at any time of the activity to display additional information about each organ. Users can use those hints to infer the correct sequence of tangibles and learn more facts about the function of each organ.

Generalizable Design Principles

During the design process and user testing of EarExplorer, we learned important lessons that can be useful to other designers. The most critical issue was to deal with the tension between giving students enough freedom to explore a domain, and constraining the activity in a way that makes it doable for most students. We describe 5 preliminary principles that guided our design process:

1) Clear Goal: from our experience, students become frustrated with an open-ended environment if they don’t have clear directions to follow. In our system, students know exactly what they need to do (i.e. reestablish an auditory connection between the ear and the brain) and which tools they can use (i.e. tangibles). Note that this is different from rigorously scripting the entire experience.

2) Direct Feedback: each step needs to be scaffolded in such a way that students can verify the validity of their system. In our system, a feedback mechanism is built in EarExplorer: students can check that sound waves are propagated to the next organ before they move to the next step. In the early iterations of the system, we observed that without clear feedback, students tend to lose time, energy, and engagement, and are less likely to complete the task.

3) Production and agency: conceptual learning happens when students are not only consumers, but also active producers of an artifact; we leveraged this fact by having them recreate the hearing system, in a tangible way. At the end of our task, they could look at the tangible system they built and consider this artifact as their own creation.

4) “Just in time” resources: people, in general, are quite poor at dealing with lot of information out of context, that then needs to be applied all at once. Discovery learning systems need to provide affordances for learning contextual content just in time. In EarExplorer, students can easily use the “infobox” to clarify points of misunderstanding or help them discover the next step of the problem.

5) Allow for “Productive Failure”: EarExplorer does not value only the right solution, but allows students to fail in productive ways. We adopt a constructionist approach by encouraging students to explore the problem space by “fixing” this micro-world, rather than merely “ingesting” information from a textbook or lecture.

User Evaluation

We conducted a user study around this interface to explore its potential in a discovery-learning activity. The results are currently under review and will be added to this page when published.

Nowadays massive datasets are becoming available for a wide range of applications, with education no exception: Cheap sensors can now detect every student movement and utterance. Massive Open Online Courses (MOOCs) over the web collect every click of users taking classes online. This information can provide crucial insights into how learning processes unfold in situ or in a remote situation. However, researchers often lack the tools to make sense of those large datasets; this work proposes additional ways to explore massive log files and describe how collaboration unfolds based on gaze patterns. Eye-tracking data is of particular interest for me, because the technology is becoming ever cheaper and ubiquitous. Several eye-tracking devices are now affordable to the general public, not just to researchers, and there have been multiple interesting attempts at using regular webcams (such as the ones integrated in laptops) to perform basic eye-tracking tasks. Even though the data generated by those low-cost devices is still far from being perfect, there is a trend suggesting that their price is steadily decreasing and their accuracy improving. On the long run, it’s likely that every single device found in the market will be equipped with some kind of eye-tracking technology.

The dataset

I previously conducted an experiment where dyads of students (N=42) remotely worked on a set of contrasting cases. The students worked in pairs, each in a different room, both looking at the same diagram on their computer screen. Dyads were able to communicate through an audio channel over the network. Their goal was to use the displayed diagram to learn how the human brain processes visual information. Two Tobii X1 eye-trackers running at 30 Hz captured their gaze during the study. In the “gaze” condition, members of the dyads saw the gaze of their partner on the screen, shown as a light blue dot, and they had the opportunity to disable this overlay by pressing a keystroke (interestingly, none of the students chose to deactivate the gaze awareness tool); in the control “no gaze” group, they did not see the gaze of their partner on the screen. Dyads collaboratively worked on this task for 12 min; they then read a textbook chapter for another 12 min. This text provided them with explanations and diagrams about visual processing in the human brain. The structure of the activity followed a PFL (Preparing for Future Learning) type of learning task (i.e., contrasting cases followed by a standard instruction). Students finally took a post-test and received a debriefing about the study goal. I found that this intervention—being able to see the gaze of their partner in real time on the screen with the gaze awareness tool— helped students achieve a significantly higher quality of collaboration and a significantly higher learning gain compared to the control group. Additionally, the two eye-trackers running captured students’ eye movements during the study and stored these data as logs; because of technical issues, we only have the complete eye- tracking data for 16 pairs (N=32).

The contrasting cases that students had to analyze

Goals

This work has several goals. The first is to provide an alternative approach for exploring eye-tracking data, involving data visualization techniques. I conjecture that uses of visualization techniques for representing massive datasets can provide interesting insights to re- searchers. Previous work has sought to develop visualizations for representing dyads’ moments of joint attention (cf. the cross-recurrence graph below); I want to propose an alternative and perhaps more intuitive way of visualizing this particular kind of data, e.g., by building networks that represent students’ shared visual attention. The second goal is to compute network measures based on those graphs, so as to examine whether some metrics are significantly different across the two experimental groups. Those metrics can provide interesting proxies for estimating dyads’ quality of collaboration. Finally, I tried to automatically predict students’ quality of collaboration by feeding network features into machine learning algorithms.

Cross-reccurence graphs are the standard way of visualizing dual eye-tracking data. The x-axis shows time for subject 1, while the y-axis shows time for subject 2. Dark points on the diagonal represent moments of joint visual attention (JVA): group 1, on the left, exhibits low levels of JVA, while group 2 on the right is highly synchronized.

Using fixations as nodes and saccades as edges in a network

To construct graphs from gaze data, I divided the screen into 44 different areas based on the configuration of the diagrams learners were shown during the study. Students had to analyze five contrasting cases; the answer to the top left and top right cases were given. Possible answers were given on the right. Students had to predict the answer of the three remaining cases. I thus segmented the screen into squares, which provides me with 30 areas that cover the diagrams of the human brain and 8 areas that cover the answer keys. In our approach, edges are created between nodes when we observe eye movements between the corresponding areas of interest. The weight of an edge is proportional to the number of visual transitions between the corresponding screen end-points. A first (unsuccessful) attempt used individual as the unit of analysis for the graph. Those networks were too dense, and too highly connected to be useful. The next attempt involved building one graph for each dyad. Here, I wanted to capture the moments in which dyad members were jointly looking at the same area on the screen. The nodes correspond to the screen areas, and edges are defined as previously (i.e., number of saccades between two areas of the screen for an individual).

Networks built with eye-tracking data. The graph on the left shows a group with a high quality of collaboration; the graph on the right shows a group a low quality of collaboration.

From a data visualization perspective, this approach conveys key patterns in collaborative learning situations. The top left graph above shows a dyad in the “no-gaze” condition; one can immediately see that these students rarely shared a common attentional focus; nodes are small and poorly connected. The graph on the top right represents a dyad in the “visible-gaze” condition and is a strong contrast to the previous example: here students are looking at common items much more frequently and those moments of joint attention provide opportunities to compare diagrams. Nodes are bigger and better connected.

All the networks generated from the current dataset

Based on this new dataset, we computed basic network metrics. The variables below satisfied the parametric assumptions of the analysis of variance that we used (i.e., homogeneity of variance and normality). We found that in the visible-gaze condition, there were significantly more nodes (F(1,30)=8.57, p=0.06), with bigger average size (F(1,30)=22.15, p<0.001), more edges (F(1,30)=5.63, p=0.024), and more reciprocated edges (F(1,30)=7.31, p=0.011). Those results indicate that we can potentially separate our two experimental conditions solely based on network characteristics. Furthermore, several measures were significantly correlated with the groups’ quality of collaboration (see the rating scheme by Meier, Spada and Rummer; 2007): the average size of a node was correlated with the overall quality of collaboration (r (32)=0.62, p=0.039), as well as all the sub-dimensions of the collaboration quality rating scheme. Other metrics were correlated with various metrics of the graphs (for more details, see Schneider & Pea, 2014). Finally, we used those metrics with a machine learning algorithm and found encouraging results when predicting students’ quality of collaboration (again, for more details see the paper referenced below).

Conclusion

Those preliminary results show the relevance of using network analysis techniques for eye- tracking data. In particular, I found this approach fruitful when applied to social eye-tracking data (i.e., a collaborative task where the gaze behaviors of each member of a dyad are recorded simultaneously and made visible to the other member). In summary, this work provides three significant contributions. First, I developed new visualizations to explore social eye-tracking data. Researchers can take advantage of this approach to discover new patterns in existing datasets. Second, simple network metrics might serve as acceptable proxies for evaluating the quality of group collaboration. Third, I fed network measures into machine learning algorithm, which seems to suggest that those features can predict multiple dimensions of a productive collaboration. As eye-trackers become cheaper and widely available, one can develop automatic measures for assessing the dynamics of people’s collaborations. Such instrumentation would enable researchers to spend less time coding videos and more time designing studies and exploring patterns in their data, thus providing augmentation tools that enable humans and computers to each play to their strengths in the human-machine systems for studying collaboration. In formal learning environments, such measures could be computed in real time; teachers could employ such metrics of ‘collaboration sensing’ to target specific interventions while students are at work on a task. In informal networked learning, collaboration sensor metrics could trigger hints or provide other scaffolds for guiding collaborators to more productive coordination of their attention and action.

This work won the Best Paper Award at the LAK13 (Learning Analytics and Knowledge) conference held in Belgium in April 2013.

Previous research demonstrated that joint attention plays a crucial role in any kind of social interaction: From babies learning from their caregivers to parents educating their children, teenagers learning from school teachers, students collaborating on a project or for any group of adults working toward a common goal, joint attention is a fundamental mechanism for establishing common ground between individuals.

The goal of our work is to develop new ways of supporting the establishment of perceptual joint attention (as distinguished from cognitive, or social joint attention). We use eye-tracking technologies to share users’ gaze during collaborative learning.

In a unique application of eye-tracking technologies, we propose their use to inform a collaborator about their partner’s gaze during a collaborative learning situation by creating a new real-time perceptual data stream overlaid on the static representation of the learning resource they are each studying. In other words, we go beyond prior research using eye tracking as a researcher methodology and representational medium for making scientific inferences about learners or collaborating learners, to use eye tracking for providing a new real-time information resource for learners to exploit for enhancing their own collaborative processes.

Students worked on a set of contrasting cases to explore how the human brain processes visual information (each student had access to a different answer)

The experiment

Our experiment had three distinct steps: during the first 12 minutes, dyads worked on 5 contrasting cases in neuroscience that were represented in a single static diagram. They had to collaboratively explain how visual information is processed in the human brain by studying the models described in Figure 1. In the treatment group, they could see the gaze of their partner on the screen. In the control group, they could not. They then read a text on the same topic for 12 minutes. Finally, they answered a learning test with questions on the terminology used, concepts taught and questions in which they needed to transfer their knowledge to a new situation.

Results on the pre/post-test on each condition (dyad-gaze = students had access to the gaze-awareness tool; dyad-nogaze = control group). The x-axis shows the different dimensions of the learning test.

In one condition, subjects could see the gaze of their partner on the screen as it was being produced. In the other, they could not. Our results reveal that this simple intervention was associated with subjects in the first group producing a higher quality of collaboration and learning more from the contrasting cases. In particular, subjects characterized as followers saw their learning gain dramatically increase. This result was partially confirmed by a similar pattern found for students’ cognitive load: followers in the control group spent more effort than leaders while learning less; followers in the treatment group spent less effort than leaders but learned more. We also found that subjects in the “no-gaze” condition spent more time on cases 1 and 3; this suggests that they took more time (and probably had more difficulty) sharing their answers. Participants in the “visible-gaze” condition had a higher percentage of joint attention, which proved to be a significant mediator for learning.

Conclusion

It is well established that joint attention plays a crucial role in any kind of social interaction. Our study provides additional evidence that its role is also preponderant in collaborative learning situations. We predict that in a near future, eye-trackers will become increasingly cheaper and widely available to a broad range of devices (e.g., not only desktops and laptops, but also smartphones and technology-enhanced eye glasses). Our study shows that in some technology- mediated interactions, real-time mutual gaze perception is beneficial for collaboration. Those results have important implications, especially for e-learning environments, since achieving a good remote collaboration is particularly challenging. Thus, we believe that it will be promising to explore the conditions under which students’ visual exploration should be made available to their partners when working remotely. One caveat is that this awareness tool seems to work well for dyads; we are more skeptical of the use of a gaze- awareness tool for triads or large groups where this visualization may become distracting. Future work should investigate whether this effect generalizes to different tasks and group sizes. Our findings also have indirect implications for co-located interactions; as Barron (2003) highlighted in her study, having students collaborate in the same space, either side-by-side or face-to-face, does not make the establishment of joint attention trivial. We hypothesize that our intervention may lead to similar benefits for students working on an interactive surface (as while wearing eye-tracking goggles). Finally, our results have further implications for teachers’ practices; with training, we posit that gaze-awareness tools could teach students the value of achieving joint attention in collaborative groups. The ability to effectively collaborate with peers was recently highlighted as a crucial 21st century competency.

Teaching abstract concepts is notoriously difficult, especially when we lack concrete metaphors that map to those abstractions. Combinatorix offers a novel approach that combines tangible objects with an interactive tabletop to help students explore, solve and understand probability problems. Students rearrange physical tokens to see the effects of various constraints on the problem space; a second screen displays the associated changes in an abstract representation, e.g., a probability tree. Using participatory design, college students in a combinatorics class helped iteratively refine the Combinatorix prototype, which was then tested successfully with five students. Combinatorix serves as an initial proof-of-concept that demonstrates how tangible tabletop interfaces that map tangible objects to abstract concepts can improve problem-solving skills.

Introduction

Many decisions benefit from understanding probability, e.g., when a patient must interpret the meaning of a medical test result or when a politician must weigh the costs and benefits of a particular policy. Unfortunately, Tversky and Kahneman demonstrated that everyone, even professional statisticians, suffer from systematic biases in their intuitive judgements of probability. Students make a variety of identifiable mistakes when solving probability problems and even graduate students who plan to teach mathematics retain strong misconceptions.

The challenge is how to help students develop an intuitive grasp of these abstract concepts. We are particularly interested in combinatorics, a branch of probability that deals with the enumeration, combination, and permutation of sets of elements and their mathematical relationships, because it results in a combinatorial explosion: even simple problems result in hundreds of possibilities that cannot be represented simply with physical objects, virtual or otherwise.

Design Challenge

The original motivation for this project stemmed from observations of students in a university-level course in combinatorics. Faced with only paper and pencil, many had difficulty developing intuitions about probabilities and suffered from the ‘stereotype threat’ that they are poor in math. We hoped that letting students manipulate concrete objects while simultaneously observing the corresponding changes in deep structure, e.g. a probability tree, would reinforce their intuitions about the underlying mathematical principles. Our goal was to create an engaging and playful environment that avoids excessive mathematical notations and encourages discussion.

Combinatorix

Hardware

Combinatorix (Fig. 4) supports several input techniques: a camera detects the location of fiducial markers and a wiimote provides the position of multiple infra-red pens. A projector displays additional information around the tangible objects. The interactive surface is 60 x 45 cm. and can accommodate up to four students at the same time.

The Combinatorix setup: The webcam detects location of fiducial markers; the wiimote detects position of infra-red pens

Software

The underlying application is written in Java and uses the Reactivision engine to detect fiducial makers [7]. Additional libraries, e.g., wrj4P50, communicate with the wiimote. The system is modular and can easily accommodate the creation of additional operators for constraining the sample space.

The current version displays two kinds of information: first, the tabletop interface shows a specific number of placeholders for objects. Letters can be placed on those spots to form a new combination. At the same time, the remaining number of letters for each step is displayed on top of each placeholder. A second screen displays a probability tree reflecting the current state of the problem. Letters can easily be replaced by other elements, including virtual, laser-cut and 3D-printed physical objects. Combinatorix supports up to 10 tangible objects and 20 virtual ones.

User Study

We contrast two educational positions in our user study. The first one, a “tell-and-practice” approach, advocates direct instructions followed by practice exercises. The idea is to expose students to the “truth”, and then reinforce this first exposition with drilling exercises. The second approach (labeled “inventing”) suggests providing carefully designed activities to activate prior knowledge in students, which can be then confronted with experts’ explanation of a domain. The idea here is to have students formulate their own theory of a phenomenon, and then have them realize the many subtleties that differentiate their basic understanding of a concept with expert theories. The first approach is widely used in classrooms, while many researchers in the learning sciences advocate the second one.

Experimental design of the user study

We computed learning gains by subtracting students’ scores on the pre-test from their scores on the post-test. The scores in the post-test supports the main hypothesis: Students who completed a hands-on activity on an interactive tabletop and then watched a mini-lecture significantly outperformed students who first watched the lecture and then completed the hands-on activity: F(1,22) = 9.28, p < 0.01, Cohen’s d = 1.61 (mean for the “video-table” group = 2.23, SD = 1.77, the “table-video” group = 4.23, SD = 1.42). This effect remained significant when computed at the dyad level: F(1,10) = 7.73, p < 0.05.

Students’ scores on the pre-post tests

Conclusion

Our findings suggest that innovative technologies can have radically different effects on students’ learning depending on how they are integrated with traditional teaching practices. Choosing the wrong sequence of activities may impede students’ learning, whereas adopting a constructivist perspective is likely to foster knowledge building. In this study we were able to replicate previous results showing that TUIs increased students’ learning gains when used as a discovery-learning tool before traditional instruction (as opposed to a “tell-and-practice” type of instruction). Those results have implications for a wide range of educational approaches (e.g., classroom instruction, flipped classrooms, MOOCs): when correctly designed and implemented, TUIs can boost students’ learning by: preparing them for future learning, providing them with a fertile ground for socio-constructivist activities, supporting their exploration of a problem space, and increasing their engagement in hands-on tasks.

Acknowledgments

I would like to thank the Amir Lopatin Fellowship for funding this project; I would also thank Wendy MacKay and Paulo Blikstein for their support of this project.

Multi-Surface Multi-Touch Simulation of Climate Change and Species Loss in Thoreau’s Woods.

Translating scientific findings into publicly understandable forms is an important step in making science accessible to the general public, thus is crucial in the public awareness of our environment, health and well-being. Visualization can be a powerful tool in achieving this translation. The goal of our research is to use interactive visualization coupled with predictive simulation as a fun and engaging informal learning tool for public informal science education. In this paper, we present WALDEN, an interactive visual simulation as a case study to examine whether a large multi-display multi-touch platform is appropriate for this type of visual informal science education. Our research focuses on the design of an interactive visual simulation system for illustrating and predicting the impact of climate change on the phylogenetic (evolutionary) patterns of species loss in Thoreau’s Woods in Concord, Massachusetts.

Design Requirements

Visualization designed for informal science education audience needs to be illustrative of the core scientific concepts, and also be self-explanatory for non-biologists. For our particular scientific story, we need to (1) present multiple related concepts and parameters including phylogeny, phenology, warming temperatures, and seasonality, and (2) provide visual illustrations such as imageries of flowering plants, geographic distribution of these plants, and the changing climate. To accommodate this large visual content requirement, we employed a multiple display environment that included a multi-touch tabletop (Microsoft Surface) and a large rear projected display wall of 12×6 ft2 (3073 by 1536 pixels).

Two users interacting with WALDEN.

The WALDEN simulation seeks to model changes in abundance of a select group of plants found in Concord, as a response to climate change and human intervention in the local ecosystem. The stochastic simulation uses empirical plant abundance and climate change data observed in Concord over the last ≈100 years, to make future projections on the floral populations in this area.

The abundance change simulation is broken down into three modules: temperature shift (how individual species of plants are able to shift their flowering time (ft) to match fluctuations in annual average temperature; not evolutionary conserved), seasonality (how species are able to respond to fluctuations in temperature from year to year, measured as the standard deviation of average temperatures between years; phylogenetically conserved), and re-growth (the ability of plants to regenerate their populations to their initial equilibrium levels). Each of these modules acts as separate, individual processes that affect the change in abundance of the individual species of plants.

The two screen interfaces of WALDEN.

We carried out two rounds of iterative designs to arrive at the visualization and interaction described. The visualization in Figure 2 illustrates how simulated flower populations are affected by changes in annual average temperature and seasonality. People interacting with the simulation can alter the temperature and seasonality values directly on the multi-touch table, and the effect of these changes are presented on the detailed simulation overview on the large wall display.

Datawall: (A) Cross the top of the data wall, a dynamic scrolling graph displays the simulated annual average temperature (shown as a black graph line), which is superimposed over the simulated seasonality of that period (yellow band). The abundance values of the currently selected species are shown below. Both graphs use a logarithmic scale on the x-axis in order to visualize a longer time period. (B) On the bottom left is a radial phylogenetic tree of the 429 floral species from the Concord area [6]. The simulation only models a subset of these species, and these nodes are identified within the tree by a label presenting the name of the flower and its current abundance value. The currently selected plant species is highlighted in red. (C) On the bottom right, additional information about the currently selected plant species is presented: a map of the species distribution throughout the USA, a scientific drawing of the species, and an example photo image of the flower.

One interaction technique used on the tabletop.

We conducted a qualitative evaluation. The main objective was to assess potential learning gains. In total, 10 users (6 individuals and 2 pairs) participated in the study. We used a think-aloud protocol for prompting comments from individuals. At the beginning of each session, users read a short introduction on phylogeny, and the abstract of Willis’ paper. Our analysis suggests that every user grasped the main idea that closely related plants tend to react in a similar way to climate changes when using the system. Selected quotes from the participants illustrate this understanding: “I notice that similar species seem to be affected in the same way by similar stimulus”, “Species that are close through ancestors have similarities in the abundance behavior”, “Closer species were affected quite similarly by changes in temperature”.

We also noticed that the setup of tabletop/datawall effectively separated action and reflection: users tended to build hypotheses based on their existing pre-conceptions, make changes on the tabletop and then reflect on the results displayed on the large vertical display. Because users had to wait around 10 seconds to see the effect of climate change on the plants, they took advantage of this time to predict the plants’ behavior and elaborate alternative hypotheses.

Thanks to Matthew Tobiasz, who built the system; Chia Shen, who funded and supported this wok; Charlie Willis, who provided the scientific evidences on which this project is based; Laurence Mueller, for his moral support.

This project was part of a d.school class taught at Stanford (d.science). Here is the class blog. The goal of this project was to provide ways for students to share their level of understanding or confusion during a class lecture. We developed several prototypes of a phone-cliker which is a kind of “awareness tool”; students can see in real time how confused the class is. This facilitates action taking (stopping the professor, asking for clarifications) and community building. The following video shows how one of our prototype would work in a real classroom:

USER GROUP

Freshmen taking science classes (CS, math, physics…) in college.

EXISTING WORK

There is a general trend of ACTIVE / PARTICIPATORY learning for cognitive engagement in the classroom:

Clicker / Personal Response systems widely used

Also some clickers that communicate student understanding to teacher, but not necessarily to other students

Nothing addressing the stigma of asking a strange question in class… nothing helping students to understand each other’s collective knowledge or to engage the teacher in cooperation

existing systems are generally expensive; no low-cost solution to this need

Current work aimed at increasing teacher feedback is primarily focused on increasing student cognitive engagement. The intent of these systems is to keep students engaged by answering questions, with a secondary benefit of giving the teacher feedback on in-the-moment student understanding.
However, these systems fail to address two needs: the needs of students to feel safe and not judged for actually asking a question by raising their hand during class, and the need of the teacher for post-lecture data about student understanding.

EMPATHY OBSERVATIONS

Most of our observations come from a computer science class (“probability for computer scientists”). We observed TA’s office hours / lectures / interviews with students. Large lecture hall with hundreds of students. In situ, totally appropriate location and user groups. We were not intrusive. Students in lecture were watching and taking notes or surfing the web; sometimes students ask overly-advanced questions that aren’t helpful for the class. Students in office hours were working, getting help from the TA, or waiting.

SYNTHESIS

STUDENT 1

“There is a crucial moment when you learn a new formula, and you have to stop listening to the professor in order to mentally compute each steps.”

“Lack of big picture; you’re just flooded with information and you need to recall as much stuff and procedures as possible.”

“Video tapping is helpful because you can stop and do some consult other resources when needed.”

“I love when the teacher says a joke after a complex equation, because it gives me time to conceptualize it.”

“A huge cognitive load comes from the fact that you have to mentally replace the indices and variables in addition to knowing what they represent; writing their name in terms of the problem (concretely) helps a lot.”

TA

“Students usually get it when I explain a concept to them; at least for a moment. The insight often doesn’t last.”

“I try to make problems more familiar to students by using concrete examples, or by breaking it down in smaller problems.”

“It’s difficult for students to generalize from one example; you change a small thing and they lose it.”

STUDENT 2

“The class is very useful but also painful. I feel like I’m really spending a lot of time on it.”

“I need to look at the slides 3-4 times before class and again 3-4 times after class to feel like I have understood the lesson.”

“Students have to learn to ask conceptual questions to the TAs; office hours have become sections because there is too much content in a lecture.”

“It’s easier for people who are always doing maths; for others, there is a huge learning curve.”

OTHER STUDENTS

“I never raise my hand.”

“I’m slow, so it’s difficult to quickly come up with a good question.”

“I’m okay asking questions 1 on 1, but I don’t do it even in small groups.”

“Often I have a basic conceptual question and I don’t want to ask it because I should know the answer.”

“It would be helpful to know what the rest of the class thinks.” Repeated by all students!

INSIGHTS

Students generally aren’t comfortable giving direct performance feedback to the teacher, especially during class; they want to do it privately and anonymously to save face for themselves and the teacher.

Students are more comfortable asking questions if they know that others have the same question.

Students are more comfortable asking questions if they know the other students personally.

It’s hard to speak up during a lecture for two reasons – classroom culture, and size / anonymity.

Points of View

POV 1

INSIGHT: People joke about not having time to write stuff down, but they don’t ask the teacher to slow down!

NEED: student needs a way to feel better about asking the teacher to slow down without pissing off other people or looking dumb.

POV: “I don’t want to be ‘that guy’ in class who is holding everyone back – I want to do something that helps me but is also good for everyone else.”

POV 2

INSIGHT: Teacher doesn’t realize that they’re going too fast.

NEED: Feedback from student. A way to create a classroom culture in which it’s okay to ask the teacher for things.

POV: “I’m doing the best I can for my students, and I don’t want any assumptions to hold us all back from creating a great learning experience.”

BRAINSTORMING SOLUTION

We want to come up with a way to have students loose their inhibition towards asking the professor to slow down, and this will probably happen if they realize they’re not alone in that desire. Let’s give students a way to discreetly let the rest of the class know that they feel rushed and want the lecture to slow down; clicker systems have been used for similar feedback before, so let’s make one with that goal.

PROTOTYPE 1

NEED: student needs a way to feel better about asking the teacher to slow down without pissing off other people or looking dumb.

IDEA: Give students clickers so they can show they want the professor to slow down; then show the dynamic results of this to all the students during the lecture, to show they’re not alone in that desire.

VARIABLE: Will students ask the professor to slow down if prompted by a passive indicator?

We asked students what they thought of this feedback mechanism and concept, but did not test it in a real lecture setting.

INSIGHTS

Students liked and wanted to use the feedback mechanism; it was like a game.

Students said they would be encouraged to give feedback, but if no one does, the professor still does not know about class confusion.

If students have their phones out, will be tempted to goof off? Prevent students who leave the app from giving feedback?

Is a generic indicator of “confusion” enough information to convey to students about their peers?

What about students who don’t have smart phones?

Students might down-vote difficult sections that are still necessary. They are not the best judges of learning content.

Feedback isn’t constructive. This puts a lot of demand on the teacher to be a good performer; will this frustrate teachers who get bad feedback?

PROTOTYPE 2

NEED: teachers need to refer to previous feedback to identify complex concepts or slides that are confusing, since students are not giving the lecturer the feedback in-class.

IDEA: Build an histogram of students’ confusion (updated over time as class goes on).

VARIABLE: Will the lecturer use the immediate feedback to help with explanation?

Based on insights from previous user testing, we choose to improve our previous prototype “clicker” system to include a way to explicitly give the professor immediate feedback. Students still are able to press a “help” button on their clicker or phone and a program records when during the class this button was pressed.

We show the professor, in lecture, a graphic that shows “student confusion” as a function of time. They can see how well understood their comments just were, and it allows them to go back after class and look at what parts of their presentation need to be clarified and improved, by looking at what timestamps had the most confusion.

We tested this system in a small lecture given by one of David’s labmates. The lecturer could see a plot of when during his talk students were having the most confusion; the audience had no feedback, other than knowing when they themselves pressed “help” on the clicker page.

Histogram of students’ confusion during lecture:

Line chart directly next to the slides:

INSIGHTS

Audience felt guilty about using clickers: “I should just raise my hand, not use this thing.” Even after one testing session, we might be changing classroom culture!

Attendees did not use clickers very much. A class-facing component seems to be necessary to keep faith and engagement in the system.

Presenter did see chart clearly, and tried to act upon the new information.

“I didn’t know what to re-explain.” Presenter though that this would be more useful after lecture, when he would have time to re-assess the specific trouble spots.

Prototypes need to be tested in environment that mimics the intimidating environment of a large intro class; we were in a smaller, optional lecture, that was very low stress.

Presenter might need training as to how to deal with immediate feedback usefully.

The system needs both prototypes simultaneously to be functional; lecture environment can be changed more if there is an audience-facing screen, lecture confusion can be reduced more if there’s a professor-facing screen.

PROTOTYPE 3

NEED: Teachers needs to be able to identify what was confusing during their lecture (a histogram may not be enough); name of confusing concepts.

IDEA: Allow students to directly type a keyword on their phone / laptop

VARIABLE: Will students be willing to document confusing concepts? Can the lecturer incorporate current confusing topics into their talk in real-time?

We had previously ignored how the lecturer would react to this feedback, but now that our last prototype gave us specific feedback on that, we decided to incorporate more informative feedback into the system. We still wanted to keep the mechanism by which the students asked for help very straightforward, so we show a list of most frequent topics on people’s phones and allow students to easily type in new topics at the bottom. The current most requested topics are shown on the presenter / class-facing screen along with the previous histogram data.

We were unable to find an appropriate lecture that was willing to let us test out our clicker system before the end of the quarter; based on our previous testing, we knew that we needed to find a class that was as close as possible to the original lecture situation we performed needfinding on. Unfortunately, the larger lecture classes were unwilling to introduce things that might delay class this late in the term. We instead asked people and presenters for feedback on this updated system.

INSIGHTS

Presenter still was able to interpret data, and said he could incorporate it into his responses easier.

Presenter said that this would be especially helpful in reviewing how a presentation went after the fact, and knowing what topics to focus on / review in later lectures.

Similar to how people use hashtags during conference talks on Twitter; this shows the principle could be well received.

Now that it’s possible to directly and anonymously communicate with the professor, are we preventing classroom culture from changing? Will people be less likely to ask questions now that students are aware the professor already knows their question?

FINAL THOUGHTS

Testing that any device truly changes classroom culture will be difficult and require a longer term study than we were able to complete. We are still hopeful that further testing and answering some of our outstanding questions on the interplay between social norms between student and teacher, and between students will let that goal be achieved. The fact that after one testing session one student already said, “I should just raise my hand, not use this thing,” leads us to think it’s possible. In a sense, we’d like our project to be self-destructing; we think that any system that encourages students to ask questions and succeeds will positively reinforce the act and, hopefully, will make itself unnecessary.
Team: Daniel Green, David Selassie, Bertrand Schneider

Motivation: Wouldn’t that be great to instantaneously visualize past events that happened exactly where you are? With current technologies, this is now possible. The GPS of your phone can be used to link your position with a database loaded with important historical events related to your location. We believe that this kind of casual, on the go learning is being facilitated with the technological advances we are witnessing.

More specifically, the tracking of movement across place and the embedding of narrative within place enabled by GPS-enabled phones and object-aware systems offers the opportunity to collect data related to fundamental questions raised by philosophers for thousands of years about how humans create, perceive and interact with their worlds. Social scientists have been interested in how cultures construct a sense of place, and the role that place plays in meaning-making, cultural cohesion, and sense-of-identity within communities. Examples include Keith Basso’s influential work on the role of place-based storytelling in inter-generational knowledge sharing in Apache culture; the role of place-based narrative in the maintenance of national identities engaged in conflict (e.g., Israeli and Palestinian national identities formed in relation to the city of Jerusalem), and identities of displaced or re-placed peoples such as refugees and migrant workers. These issues are not limited to those who are marginalized; as our work lives become increasingly mobile, flexible and virtualized, there is common awareness that the human sense of place is changing. However, untill recently researchers lacked access to large datasets related to people’s movements, real-time sense of place, and production or consumption of media about place-based experiences – instead having to rely on post-hoc self-report measures such as interviews and surveys. Research has shown that the cognitive and social resources embedded in places significantly influences what we think about, how we learn, and what we choose to do and learn next. While research shows that place is a large factor in the formation of social identity, little is known about how transition among places influences learning, and how geo-locationally aware tools might change or enhance perceptions of and movements through places. For the first time in history, we have the possibility of accessing the movements and real time place-based thought and learning patterns of communities.

TimeExplorer provides the kind of functionality described above. Students are able to retrieve stories related to their location on their IPhone; they can also post new stories and comment on them. See the screenshots bellow for more details,

As well as a demo of the app:

Research Team: Bertrand Schneider, Shima Salehi, Sarah Lewis, Roy Pea

Status: the app has been removed from the app store due to the expiration of my developer licence (16 Feb 2012)

This project was part of the Beyond Bits and Atoms class taught at Stanford by Prof. Paulo Blikstein. We had to find a child, interview him/her, discover what their dream toy would be and build it. Below I describe my process and various iterations of prototypes:

My interview with the kid went well, I tried to alternate between asking her questions and playing with her. We spend one hour in her room, where she described me all her toys and how she played with them. She loves zebras, and had a big collection of them: big, small, soft, hard, etc. Her dollhouse has even been converted into a zebra house, and the zebras actually sleep in standard beds with humans.

She described me the toy of her dreams as a box where zebras and giraffes could race against each other. Her design included several elements:

a field of grass where the animals could race

she wanted to have 10 zebras and 10 giraffes

the victorious animal would win a zebra cup

she mentioned teams that she could switch so that the other animals could rest

We conducted this part of the interview in the living room, where her mother helped her describe the toy she wanted. It was actually very helpful to have one of her parents with me, because information flew more easily between them (and obviously the mother knew more easily what the kid meant). She also drew several sketches for me:

she decided to draw this zebra and giraffe so that I could use them for her toy. She also drew a field of grass, with the starting point on the left and the finish line on the right:

Here is the part where she describes me her toy:

I tried to follow her instructions as closely as I could but I quickly realized that I wouldn’t be able to provide her 10 zebras and 10 giraffes. My first (and biggest) challenge was to make the animals move in the running lanes. My first attempt involved a treadmill coupled to a gogo board:

That didn’t work so well, because the paper would easily slide outside the wheel. I could have solved this problem by cutting and gluing some acrylic circles to the sides of the spools; however the whole system was too fragile and too complicated to build. So I changed my mind and tried to use rubber bands to propel the animals:

This first prototype didn’t work very well, mainly because the wheels were too small and the force wasn’t evenly distributed over time (the system above would use all the force in one shot). With the help of a TA, we designed another iteration that successfully propelled our car frame:

The rubber band is attached to the frame of the car on one side, and on the axis of the wheels on the other side. By moving the car backward, the rubber band gets twisted around the axis and accumulates force; by releasing the car, it discharges its tension.

The final design of the boxes involved a zebra texture on the outside and multiple decorations on the inside: some seats so that other animals could watch the race, a zebra cup after the finish line and an inscription on the ground (“zebra race”):

Here is the final video of the project:

The kid was very happy with it, and we spent quite some time playing with it afterwards. Among other remarks, she said that she didn’t think that I could make the toy that good, and that the box was “the goodest toy she ever had”.

In this project we discuss the design and evaluation of a gaze-based interaction technique for visualizing a phylogenetic tree. In our implementation, users were able to brush, expand, collapse and rename nodes of the tree by looking at the species of interest and using the keyboard. Our goal was to investigate the effectiveness of gaze-based interactions for a specific task (e.g. exploration of tree structure).

we tried different visualizations and discovered that natural and efficient gaze-based interactions are difficult to implement. Our original idea was simple – to use gaze to facilitate users’ perception of a graph. We originally implemented three prototypes enhanced with gaze-based techniques: a scatter plot, a bar graph, and a choropleth map. We wanted the detailed information to be revealed when users focused on a certain region, whereas things in the periphery would stay hidden. We soon realized that this approach was not ideal for users, as having the detailed information shown on a single region prevents them from comparing or analyzing content on other parts of the visualization. Moreover we observed that having the visualization radically change based on the gaze’s location is highly distracting. We concluded that users would prefer to have everything displayed because they are able to choose what they want to see without bothering moving their gaze.

We compared this approach with a purely mouse-based interface. We found the following results:

Users were significantly slower to perform the “brushing”, “navigating” and “analyzing” task when using their gaze. Users were equally fast for the “renaming” task when using a mouse or their gaze. There are several reasons for this pattern of results: the inaccuracy and latency of the eye-tracking technology we used, as well as a certain discomfort for users to control things with their eyes.

Even though eye-tracking techniques can potentially compete with traditional input devices, users still consider those interactions as more demanding. It is not clear whether this feeling would decrease over time with appropriate training. In conclusion, we believe that eye- trackers should be used in conjunction with other interactions techniques for simple tasks (such as switching between windows or selecting a text field). Trying to perform complicated tasks with the gaze is similar to control the rhythm of our breadth: this is a conscious and effortful task. As a conclusion, designing efficient gaze- based interaction is complex and difficult; researchers have not found yet a situation where eye-tracking techniques offer a real advantage over traditional input devices.

As future work, we are interested in developing less disruptive interaction techniques for eye-trackers. More specifically, actions that don’t require visual change on the screen.