Using Evaluation to Shape ITS Design: Results and Experiences with SQL-Tutor

View/Open

Author

Date

Permanent Link

This paper presents the results of three evaluation studies performed during 1998 and 1999 on
SQL-Tutor, an intelligent tutoring system for the SQL database language. We have evaluated the
system in the context of genuine courses, and used the results to further refine the system. The
main goal of our research has been the exploration and extension of Constraint-Based Modeling
(CBM), a student modeling approach proposed by Ohlsson (1994). SQL-Tutor provided us with
experiences of using CBM, and we used it to extend the approach in several important ways. The
main goal of all three evaluation studies was to determine how well CBM supported student
learning. We have obtained positive results. The students who learnt with SQL-Tutor in the first
study performed significantly better than those who did not when assessed by a subsequent
classroom examination. Furthermore, the analysis of students’ learning shows that CBM has a
sound psychological foundation.
Besides the evaluation of CBM, we also evaluated the improvements in terms of student
assessments of the usefulness of the system and evaluated various techniques used in SQL-Tutor.
In the second study, we evaluated the effectiveness of feedback provided to the students. This
study showed that high-level advice is most beneficial to students’ learning. The focus of the
third study was different. We extended CBM to support long-term modeling of student
knowledge, and used this extension to develop an adaptive problem-selection strategy. The study
revealed the benefits of this strategy in comparison with a simple heuristic strategy. We also
reflect on our experiences in evaluating SQL-Tutor.