LASI13 Friday: Plenary, group work and feedback

Open platforms for pedagogical innovation

Speaker: Piotr Mitros (Chief scientist, edX)

(Piotr was very impressively changing the geometry of his slides with ImageMagick at the command line just before starting his talk.)

There’s been a vision of creating something like an app store for different building blocks you could use in courses. Tremendous numbers of interactives, video players, learning resources you could build courses out of. Lots of learning analytics that live in their own space, would be nice if could use in common format, common dashboards – the Open Learning Analytics Platform proposal. Standards like this work for educational resources.

Two examples: XBlocks, Insights – prototype framework for building common learning analytics around the same platform across different LMSes.

It’s not like LTI or Tincan that are services model. More an app store model.

First, talk a bit about EdX and what it is. Not-for-profit, started by MIT and Harvard. Prior effort was MITx. Three goals: research (understand learning); quality (improve learning); access (education 1 billion people). These are very tightly coupled. Take advantage of economies of scale. Freshman physics lecture done 5000x in the US, very inefficient. Learning analytics, big data, across more learners. MOOCs drive residential quality. Drive also the other way – to build MOOC need residential course to interact with students. The design feeds both ways, via research. Even more intricately than that – it’s here’s a course at MIT, a student at state school, at Nigerian university 13:700, here’s BostonX community centre, here’s someone’s house, over to lone person in a room. Spectrum, tightly integrated.

edX Learning sciences is a small group, from MITx platform. Small team built framework for open-ended grading (MCQs, self-assessment, peer grading, machine learning smarts). Now productised. Working on Insights, framework for open learning analytics – this talk – transitioning to engineering. Then likely to focus on crowdsourcing – learning analytics, big data, HCI, big social. They’re hiring!

Why are they doing this? In the beginning, realised he had no idea what he was doing. I have 2.5 months until the students arrive. Even if I knew what was doing, there’s no time to do it. Put something together. Component model, built around mastery learning. They’re compositional. Can build a sequence, which is a module. Whole page is a module. Whole course isn’t a module because of resource constraints. Goal to make it ridiculously easy to use. Goal was lightweight pedagogical innovations. Don’t need a sysadmin, don’t need own servers (hostable and hosted), you can’t take down the LMS. Aim to build a community, allow reuse in courses; open standard so not just in LMS.

In 2011, worked on it. The “hello, world” course got bigger and bigger. XModules became XBlocks, got smaller again, released at PyCon as RFC. Other organisations looking to integrate.

Examples of XBlocks: Biostatistics course – instructor asks how to evaluate hospitals with mortality rates, move sliders, students discover the metric doesn’t work (e.g. if more older people tend to go) looks worse on that metric despite being better. Problems where instructor walks through math. Took a couple of hours of his time to build. Big pedagogical impact (and localised), for very little effort. Normalisation problem. Self assessment example – humanities courses. Student asks a question, get a rubric to self-assess. Not grading, more monitoring and metacognition. Small amount to write, doesn’t need any more servers. Scaled to more sophisticated techniques like randomized problems. Another that prevents sight of content until some action is taken – e.g. don’t see solutions until done a problem. Another that lets you build multistage adaptive testing.

There was a missing piece – a standard way to process data. One way is offline analysis. Zach Pardos analysis correlating resource usage and success. Can we have a bigger impact? Present it to instructor, in a dashboard, make it actionable. Can we automate it – item-response theory, suggestions for things to take out, or where extra help is needed. Responsive system – analyse what the student has done and respond with a hint.

Second goal for data is like goal for system – ridiculously easy to use, modular, no sysadmin, data access without personally IDentifiable information; build a community – open standard (not just edX.)

Q: Can explain can’t take down the LMS?

Say you have a db query that takes 7h to run, don’t want that to take down the system. Want it to run separately. Can’t do hostile actions that’ll take down the system, have it sandboxed in some form.

Q: Offline analysis that researchers that are doing, to instructor dashboard, there’s like proof left for the reader. Big chasm to bridge researcher to actionable for instructor. Is it that simple?

It’s an open questions. Had researchers look at video analytics, find hotspots, nice publication, but can plug nicely in to the dashboard. Capture the commands, transfer it. Instructor can act on it. Can have developers come in and clean it up. Want at least for there not to be a barrier to bring it to the dashboard.

Coming there! We have read replicas of all the LMSes. Have analytics modules on all servers – can query all other modules, has own isolated data source, some can access read replicas of the databases, get events coming in; can advertise things over an SOA interface – one is a query to get at data, other is views like e.g. dashboards. Can split across servers however you want. Could break up to a core server that stores things, aux server that has good uptime and teaching analytics, and many prototype server which are only up when you keep them up. Or another way – attach modules to each component – LMS, Forums, Wiki, Big Data, connect up. Or make a data access module that les you access the components; build to make higher level things – IRT< concept map, SNA, then Recommender engine on top of all that. It’s a prototype, building it out right now, hope to get community help.

Modules are really easy to write. (Live code display.) Example of grade display module. Dummy module – Python API – one-line function returning grades. Built shared module that gets that data, takes a filesystem where it can store stuff, and a query object, bit of code to plot histogram, save it on filesystem as a png, give it a lifetime of 5min, return HTML wrapped round that image. Then can render automatically. Can also do stream processing – event handlers. E.g. count events a user has seen. (More live code shown.)

Chris: Tincan API, seems similar, data bus, with views or queries on the repository from the highlevel.

Couple of differences. If I wrote a Tincan plugin, it’s a huge specification and chunk of code. Here, just 20 lines or so does the new view. Second, can reuse, nicer integration with instructor dashboards. It can also hide things away … Tincan is specification for a service, can’t just take a chunk of code and use it. It’s the difference between an Android App Store and a Google API. Don’t have to use this runtime, could run your own.

This is the backbone of how to extend the edX platform in very lightweight ways.

Chris: What are the benefits of a published spec vs this app version, lots of other systems won’t be in Python, all that effort in libraries isn’t useful if I’m running Blackboard and it’s all in Java. With Tincan, can interoperate despite being different implementation.

Two things confounded. One, is there a spec, there isn’t yet but there will be. Second, is it a service oriented spec or not. This can work on top of Java, Blackboard or Java could stream over it. That Python could call out to whatever it wanted. For Tincan, has to be heavyweight, cumbersome. E.g. one instructor wanted to do something with one undergrad. At LAK, said had to work through IT dept to get their data. LTI and Tincan SOA approach is allowed, but this makes it easier for you. Only affects what analytics are written in, not the LMS.

Status is that it’s an early prototype, may evolve a lot – but a few early adopters. Community review. Being productised within edX. Still need a schema/data model. We want your help! Does it meet your needs?

Q: Wondering about heterogeneity in the events you’ve seen. How nimble is this to work on Blackboard, SAKAI, Moodle, or the K12 space where it’s real diversity. Services strung together from various … Many companies fallen apart at data integration services layer. How real is it? Junio, others trying to ID common event names.

We handle this, we haven’t addressed that problem hardly at all. Have a process to address that. At edX it’s very messy. User in some places, username in others. Can decorate events – e.g. with ‘actor’ (taken from Tincan spec), abstracts that away. If made that for every events, every event could be counted. Item response theory, plot problem difficulty and quality. In CMSes, try a problem once. Mastery learning, try until you get it, the original platform was open-ended, IRT fell apart. But the technology can be built through (various ways), can build a translation layer in between. We want rough consensus and working code. Still looking for consensus. Commonly-used things will bubble up.

Neil: Inside ASSISTments, lots of interactive content. What do I do to make an XBlock? Write Python code that does something. Does it run externally? I know LTI. Or are you taking in code, how do you deal with security, trust?

Ok. We’re still dealing with this issue. Have ‘codejail’ sandbox. There’s an SOA behind them. All available and downloadable as open source, on github.com/edx – all there. We’re making these decisions as we speak. Mailing list helping.

John Behrens: Language for dealing with components, structures and so forth. In the assessment community, lot of work on language. Session on Monday on evidence-centred design and common language. Recommend article in last summer’s EDM journal by me, go over universal languages for assessment in different context.

We do need common language here. Even the word assessment means different things. In LA, many things are called indicators that correlate with performance, things that are measures of performance, things that modify how people behave. Common language or lexicon would be appreciated, especially multiple fields and disciplines.

Useful, not useful?

Q: Hard to tell.

Chris: A workshop on this would be good, hack on some code.

What would be a good venue?

Chris: This, or something similar. LAK14?

EdX is just over 1y old, by then it’ll be twice as old, the tech will have progressed. By then, this’ll be a lot more evolved, we’ll have learned a lot.

Judy: We’ve been using Class2Go, come together?

No, that’d be nice to have. Oops, no! We have come together. Stanford team is 5 developers, they’ve joined us. That relationship going well. They Skype in to our analytics meetings. Have Stanford colours too, plus more fundamental work. Very impressed with that team. Young relationship, more or less just happened. Doing cross-evaluations, there’ll be up for a hackathon while I’m at EDM.

Q: You have several servers, how do you solve availability? Combining results from servers where some may be failing. Second question, how do you define the data model, can define lowest common denominator but you’d lose the best parts. Is it good to impose on learning analytics as a field what’s useful data?

Two points. How deal with uptime. Using standard web tools. On the back end, all http requests, load balancer for if servers go down. Maybe not best way to go, trying to figure out better system. There’s more to happen for scaling. Second question, I agree it’s the wrong time to define a comprehensive spec, but how I get your demographic info we’re ready to define. Other things in between, moving in a process. I’m proposing, as of a week or two ago, a sort of rough consensus, you define, I define, we converge over time. If I want to do something different I can get events your system doesn’t (need to ) handle.

Phil Winne: I don’t understand any of your code. I’m an ed psych. The data your interfaces are collecting are coarse. Any thought about digging deeper in to how learners are interacting with information before they get next document. What info do they highlight, put in their notes, how does that correlate to a glossary, etc?

Students don’t have a place to take notes. In the LMS we collect virtually everything they do. But you’re correct that … no, there are deep inferences in some ways. Places for students to take notes are things we should build out.

Phil Long: If anyone is interested in LTI development, see me afterwards. We contributed the LTI XBlock and looking to extend it out, if interested please see me.

Abelardo: As co-chair of LAK2014, like the workshop idea. But take advantage of audience LAK addresses, people engaged in educational activities, make a test for how suitable it is. Comment about the code. In fact, targeting audience channel.

Would be wonderful to have. Would love something sooner. Product team working on it right now. Influence now can have a big impact. By March the decisions will have been made. I agree with that, and some way to get involvement sooner would be good. If you’re an ed psych, if your university has a CS dept, their undergrads could build this for you.

George: Might be an option, the distributed research lab. Could be good space for online discussions, code base presentations. About a dozen academics, anyone’s invited. Get conversation started online literally next week or next month.

Next week would be better than next month.

Neil: Thoughts about stability of your code? Backward compatibility.

It’s at same place XModule was a year ago. It’s not hard to build a compatibility layer. Can’t guarantee for the future. We’ll know a lot more about that in a few weeks as it moves in to the engineering organisation. Good time to get involved to influence the API.

Observations and Charge to the Attendees

John starts. Today is not the end, but the beginning. Building a community, moving forward with what we’ve learned. We’ll get a few reflections. Then group work time to produce products to move the field forward.

George

It’s been an enjoyable week of conversations. Normalising, bring a quality of conversation together. That’s been one of the better part of the experience. Would have liked some things different for next year. Not enough of student part in to the conversation. Unfortunately wasn’t as prominent as it could’ve been. Not good enough bringing in the international community, more broadcast not network. But incredible work in putting together, funding. Program committee, very active with much work, big thanks. Going forward, real need for us to think carefully about exclusionary dimensions of the event – we want everyone here who wants to be here. Talked about SoLAR, IEDMS, want others involved too, don’t need affiliations to be an active contributor. Anybody who wants to can be a valid contributor.

What are some of the big ideas that we think analytics can contribute toward – important to focus on. Fun to get in to details, what you might do, results you’re gaining. But at the end of the day, role for LA is to navigate educational reform. Depends how radical a vision for education you want. Better than opinion and philosophy for basing decisions.

Ryan

Ditto! Great to see expansion of interest in this area of inquiry. Used to be you could get all the people interested in one room. But vast interest. Diversity and pluralism, great to bring together people from organisations arguing about this for a while, but other communities becoming interested. Learned a lot, made good connections. Some of the workshops were the best part, maybe have some remote workshops too. Would be great if people out there in TV could broadcast to us. Was a lot of fun, thanks for coming.

Dragan

Hard to add much more. Everybody’s rather satisfied, has opinions about what can be done better. We build joint events, can’t satisfy every need or difference, work hard to improve. See potential improvement with more active tasks for participants. Working groups, rather than just workshops. Charged with specific tasks, activities, problems, get a product. But given youth of the area, type of the event was good. From my perspective, some additional voices could be heard in the future. Would like to see more emphasis on areas such as statistics. Believe we too much work on analytics promoting one size fits all. Everybody has the same dashboards. Really building the same for everyone? Or individualised learning. Can learn from e.g. health, population level interventions. The next big problem, how we contribute to learning sciences.

John

Thinking back to Monday, felt like first junior high dance of the fall, not sure if they’re like you, who’ll dance. Through the week, lot of coalescing, scholarly spirit – the joy of learning and the value of data. Dedication to improving education. Wants to thanks Simon Buckingham Shum for organising LASI Locals to make it active, and people active in the Twittersphere, blogosphere, the drinkosphere (laughter), the buttonsphere.

Didn’t conceptualise this as a one-time event, but start of a movement, an ongoing effort, cross-disciplinary and cross-boundaries. Start having the conversation about what that’ll look like. Look back, and look forward. To document that – a Google doc to form the location for working groups in breakout session before lunch.

Taylor

Thank you. We’ve had lots of inspiration this week, more perspiration than we expected. And >2% butterscotch ripple. Great ideas about what’s going to happen going forward. Fill in the survey with ideas about going forward. LASI is not a conference, not an organisation. This is a space to bring together all the people critical for building field of educational data science. We started that conversation this week. I heard, started the actual work that goes in to this field. LASI primarily supported by a Gates Foundation grant to Roy Pea (who says hi). We’re going to have to write a report. But you’ve helped us with the blogs, Twitter – makes things much easier. Thinking about how this goes forward. In working groups, document things that you’ve planned this week. Secondly, LASI doesn’t end here, it starts here, and now. Not going to all go away and a year from now … What are our big ideas? What can we make progress on this year? What would the real contribution of LASI be? Starting that actual work is very much the key. In the groups, write down some of the big problems – big as you want, but also what you could do this year. How do we bring in policymakers, what are the big questions. Privacy, access, data. How can we fix the things that (block you).

Plenary feedback

John Behrens was impressed at all the work people did, the feedback. Ask all for the highlights.

Mohit

Team Canada in the doc. One theme was tension between making products and solving products, and between educators/researchers and industry. K12 educators not well represented. Lot of future collaboration.

How to communicate across disciplines? Find a problem we can solve, something concrete. Hackathon as a future event. Also, what are we trying to optimise. Coming together, grand challenge – an idea getting buy-in from platforms with much data, a common way of representing it, and open source tools for analysing it. So have a way of getting the data, and tools with examples – get it done in 6 months-1y? (Zach)

Recognise diversity in the group and field. Spending a week together good way to ID convergence and divergence. Global issues present here. Philosophy should be compulsory. Relevant data to learners, with outcome not known in advance. Transformation of society and LA role – one tool in the toolbox. LA transgresses all trad boundaries in social science research. Interdisciplinary, inter-domain. Ethics of power around analytics – build agency or remove it. Governance. Lots to do – papers, work group, joint workshops Oxford/OU/etc.

Mohit

Workshop on pitching products to schools. Talk with TeachForum America (?) to get them onboard. Direct to consumer technologies, skipping school and state, straight to students and parents – faster.

Discussion

Phil Long: For US audience. New effort under Digital Promise, interest to get input from and ideas about how it can support LA and its use in education from preschool to workforce education and all between. And ideas, look at their website, can contribute there or through me.

Ruth: Why don’t we write a white paper about learning analytics?

John B: We already had a conversation about a book to bring this community’s ideas to psychometrics community, with April. People have practical, observable things that come out of these meetings.

Simon Knight: Talked to number of people, editing Wikipedia pages for LA, EDM and related pages. Not that it’s a sales pitch, but in terms of awareness of what the community is doing, and how the literature stakes ground. It’s important, and it’s something people look at when you talk to them about it. Talk to him.

John B: Watching on blog aggregator, Twitter feed, can continue to add to the document. Fill out the surveys! People say, “How can I ever thank you?” By filling out the survey. Was awesome to meet you all. On behalf of organising committee, thanks, and we’re adjourned.

–

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.