We proposed a special interest group (80 min gathering) at CHI ’19 to think about HCI in learning and educational contexts to foster an ongoing Learning and Education community at CHI. With the recent increase in CHI submissions relating to learning (40% more submissions than previous CHI), this SIG is an opportunity to foster an inclusive dialogue on designing and studying phenomena, tools, and processes related to learning and education. This SIG brought together researchers, educators, and practitioners with three goals in mind: (1) discussing more inclusive cross-disciplinary perspectives on learning; (2) defining future directions and standards for learning and education contributions in CHI; and (3) building community across research/practice boundaries.

Demonstrates the use of Item Response Theory (IRT) to evaluate the validity of scores for a language-independent CS1 test (SCS1). We identified 3 questions which do not align with the rest of the test and 4 questions which are potentially problematic. Presented at SIGCSE 2019.

See my blog post summary of the psychometric process for evaluating and improving assessments.

Current introductory instruction fails to identify, structure, and sequence the many skills involved in programming. In this article, we propose a new theory of instruction which sequences and structures four skills critical to introductory programming: tracing, writing correct syntax, comprehending templates (reusable abstractions of programming knowledge), and writing code with templates. We developed instruction which reflected this theory. Our exploratory evaluation found that teaching skills incrementally resulted in improved completion rate on practice exercises, and decreased error rate and improved understanding of the post-test. This theory can inform future instructional design for CS1 so we focus on the incremental development of specific programming skills.

Experiences of Computer Science Transfer Students

conference paper

Harrison Kwik, Benjamin Xie, Andrew J. Ko

In Proceedings of the 2018 ACM Conference on International Computing Education Research (ICER '18). ACM, New York, NY, USA, 115-123. DOI: https://doi.org/10.1145/3230977.3231004

Publication year: 2018

To better understand the experiences, perceptions, and challenges, we surveyed and interviewed students who transferred from another institution (e.g. community college) into an undergraduate computer science program. We found that diversity in age, socio-economic status, and career aspirations, and commute burdens contributed to making transfer student experiences unique when compared to non-transfer students. This work is among the first studies to consider experiences of CS transfer students before, during, and after their transfers and has implications for improving inclusivity for a group which makes up to up to half of CS undergraduates.

An Explicit Strategy to Scaffold Novice Program Tracing

conference paper

Benjamin Xie, Greg L. Nelson, Andrew J. Ko

Technical Symposium on Computer Science Education (SIGCSE 2018)

Publication year: 2018

We propose and evaluate a lightweight strategy for tracing code that can be efficiently taught to novice programmers, building off of recent findings on “sketching” when tracing. This strategy helps novices apply the syntactic and semantic knowledge they are learning by encouraging line-by-line tracing and providing an external representation of memory for them to update. To evaluate the effect of teaching this strategy, we conducted a block-randomized experiment with 24 novices enrolled in a university-level CS1 course. We spent only 5-10 minutes introducing the strategy to the experimental condition. We then asked both conditions to think-aloud as they predicted the output of short programs. Students using this strategy scored on average 15% higher than students in the control group for the tracing problems used the study (p<0.05). Qualitative analysis of think-aloud and interview data showed that tracing systematically (line-by-line and “sketching” intermediate values) led to better performance and that the strategy scaffolded and encouraged systematic tracing. Students who learned the strategy also scored on average 7% higher on the course midterm. These findings suggest that in <1 hour and without computer-based tools, we can improve CS1 students’ tracing abilities by explicitly teaching a strategy.

Comprehension First: Evaluating a Novel Pedagogy and Tutoring System for Program Tracing in CS1

conference paper

Greg L. Nelson, Benjamin Xie, Andrew J. Ko

ACM Conference on International Computing Education Research (ICER 2017)

Publication year: 2017

Contributes a new theory of what it means to know a programming language, a novel pedagogy and computer-based tutorial for teaching this knowledge, and evidence that 1) the tutorial promotes significantly higher learning gains over conventional programming language tutorials, and 2) that these gains predict the majority of the variance in CS1 midterm grades. (description from Andy Ko’s webpage)

Skill Progression in MIT App Inventor

conference paper

Benjamin Xie, Hal Abelson

Visual Languages and Human-Centric Computing (VL/HCC) 2016

Publication year: 2016

This paper contributes to the growing body of research that attempts to measure online, informal learning. We analyze skill progression in MIT App Inventor, an informal online learning environment with over 5 million users and 15.9 million projects/apps created. Our objective is to understand how people learn computational thinking concepts while creating mobile applications with App Inventor. In particular, we are interested in the relationship between the progression of skill in using App Inventor functionality and in using computational thinking concepts as learners create more apps. We model skill progression along two dimensions: breadth and depth of capability. Given a sample of 10,571 random users who have each created at least 20 apps, we analyze the relationship between demonstrating domain-specific skills by using App Inventor functionality and generalizable skills by using computational thinking concepts. Our findings indicate that domain-specific and generalizable skills progress similarly; there is a common pattern of expanding breadth of capability by using new skills over the first 10 projects, then developing depth of capability by using previously introduced skills to build more sophisticated apps.

I analyze skill progression in MIT App Inventor, an open, online learning environment with over 4.7 million users and 14.9 million projects/apps created. My objective is to understand how people learn computational thinking concepts while creating mobile applications with App Inventor. In particular, I am interested in the relationship between the development of sophistication in using App Inventor functionality and the development of sophistication in using computational thinking concepts as learners create more apps. I take steps towards this objective by modeling the demonstrated sophistication of a user along two dimensions: breadth and depth of capability. Given a sample of 10,571 random users who have each created at least 20 projects, I analyze the relationship between demonstrating domain-specific skills by using App Inventor functionality and generalizable skills by using computational thinking concepts. I cluster similar users and compare differences in using computational concepts.

My findings indicate a common pattern of expanding breadth of capability by using new skills over the first 10 projects, then developing depth of capability by using previously introduced skills to build more sophisticated apps. From analyzing the clustered users, I order computational concepts by perceived complexity. This concept complexity measure is relative to how users interact with components. I also identify differences in learning computational concepts using App Inventor when compared to learning with a text-based programming language such as Java. In particular, statements (produce action) and expressions (produce value) are separate blocks because they have different connections with other blocks in App Inventor’s visual programming language. This may result in different perceptions of computational concepts when compared to perceptions from using a text-based programming language, as statements are used more frequently in App Inventor than expressions.

This work has implications to enable future computer science curriculum to better leverage App Inventor’s blocks-based programming language and events-based model to offer more personalized guidance and learning resources to those who learn App Inventor without an instructor.

Classifying Windows Malware with Static Analysis

unpublished

Maryann Gong, Uma Girkar, Benjamin Xie

Publication year: 2016

Adversaries are creating new types of malicious software, or malware, at an increasing rate.
The amount and diversity of malware are making classic security defenses more and more
ineffective. Our objective is to analyze multiple dimensions of Windows executable
programs. We used machine learning techniques to build and train a classifier to identify
malicious software from benign software. We find that the best mixed features classifier we
created performs over 10% better than the average performance of 57 commercial anti-virus
programs we tested against. In analyzing the performance of the commercial anti-virus
programs, we also find bias in the design decisions that sacrifices security in favor of
usability.

Replicating First Analysis of Latent Dirichlet Allocation

unpublished

Benjamin Xie, Jason Leung

MIT 6.867 Final Project

Publication year: 2015

We analyzed, replicated, and extended experiments from the original paper on Latent Dirichlet Allocation (LDA), as presented by Blei, Ng, and Jordan 2003. We empirically analyzed and compared the performance of LDA for feature reduction and document classification.

Measuring the usability and capability of App inventor to create mobile Applications

conference paper

Benjamin Xie, Isra Shabir, and Hal Abelson

3rd International Workshop on Programming for Mobile and Touch (PROMOTO 2015)

Publication year: 2015

MIT App Inventor is a web service that enables users with little to no previous programming experience to create mobile applications using a visual blocks language. We analyze a sample of 5,228 random projects from the corpus of 9.7 million and group projects by functionality. We then use the number of unique blocks in projects as a metric to better understand the usability and realized capability of using App Inventor to implement specific functionalities. We introduce the notion of a usability score and our results indicate that introductory tutorials heavily influence the usability of App Inventor to implement particular functionalities. Our findings suggest that the sequential nature of App Inventor’s learning resources results in users realizing only a portion of App Inventor’s capabilities and propose improvements to these learning resources that are transferable to other programming environments and tools.