However, there are other elements of the residential
class experience that are taken for granted and must
be re-created manually. For example, we initially
underestimated the extent to which having a regularly scheduled meeting time sets up what we call a
classroom cadence, a rhythm to the class’s interaction. We replicated this in part through weekly routine announcements to create an online equivalent
of the in-person routine.

AI may play a key role in this lesson as well. The
dynamics that create a classroom cadence are routine, predictable, and foreseeable; thus, it should be
possible to equip an AI agent with the ability to interact in a way that establishes that rhythm. AI agents
like our nanotutors may also play a role in re-creating
natural features of the residential class; the online
environment does not have a natural equivalent of a
class exercise in which an instructor can intervene
live to give feedback, but our interactive exercises
play exactly this role.

Using Automated Evaluation for
Frequent Formative Feedback
As education scales up to classes with hundreds or
thousands of students, one of the pushes is for an
increased emphasis on automatic evaluation. This
can range from simple multiple-choice quizzes to
more complex simulation-graded assignments. What
is often lost in this emphasis is the incredible influence these forms of automated evaluation can have
on students’ individual feedback cycles in working
on assignments. Many classes only run these automated evaluators after students have submitted their
work. The emphasis here is on generating grades, not
generating feedback or supporting learning experiences. If the evaluation is generated automatically,
though, it presents a wonderful opportunity to equip
students with the tools necessary to rapidly iterate in
their understanding.

As noted previously, students in the KBAI class
design agents that can answer a set of problems the
student can see, and we then evaluate them against a
set of problems the student cannot see. Prior to the
summer 2016 semester, this latter step was only conducted after the submission deadline. In summer
2016 term, however, we launched a new automated
grader that would allow students to see their agents’
results on those unseen problems without having
access to the problems themselves. This presents a
lesson to any course developing automated evaluation solutions: while it is natural to focus on such
solutions for generating the grades necessary to scale,
make sure to extend the benefits of those automated
evaluators to the students as well through frequent
formative feedback.

ConclusionsCreating and delivering the online KBAI class hasbeen one of the most satisfying educational experi-ences of our careers, and we wholeheartedly encour-age anyone with the opportunity to participate inthis new environment to try it out for themselves.The level of student motivation, engagement andownership are worth the massive time needed to cre-ate and deliver these courses. That said, there aremany open issues left to address. With regard tousing AI to teach AI, we are still exploring the rangeof topics that can be addressed by nanotutors, thelevel of authenticity that can be provided throughclass projects, as well as the development of virtualteaching assistants can answer automatically someclasses of questions on the discussion forums. Withregard to online education as a whole, we must con-tinue to explore metrics for ensuring that learningoutcomes rival or exceed residential equivalents andrepresent real value to students.

AcknowledgementsWe are grateful to several people for their support forthe development of the OMS CS7637 KBAI class: thestaff of the Georgia Tech OMSCS program, especiallyZvi Galil, Charles Isbell, and David White; the Geor-gia Tech team within Udacity, especially SebastianThrun, Jason Barros, and Jennie Kim; and the staff ofGeorgia Tech Professional Education, especially Nel-son Baker. We are especially grateful to our video pro-ducer, Aaron Gross, for producing the MOOC mate-rial for the course, and to Joe Gonzales for creatingthe peer review system we use in the class. Lastly, weare grateful to the numerous who have worked asCS7637 Teaching Assistants since Fall 2014. This arti-cle is based in part on Goel and Joyner (2016a,2016b).

Carey, K. 2016. An Online Education Breakthrough? A Master’s Degree for a Mere $7,000. The New York Times,
September 28. ( www.nytimes.com/2016/09/29/upshot/an-online-
education-breakthrough-a-masters-degree-for-a-mere-7000.
html)