Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Phil is with ed info connections in DCI am with IUMJ Bishop is the director of CIELT for the university system of Maryland

Let me summarize the four main points in our paper and in this talk.Paradigm shift in conception of data. It is a sociotechnological one.I will then argue that this movement involves four distinct fields and that together these fields might be most productive if merged into one broader notion of EDSWe then discuss five common features that unite the currently disparate EDS communities WE then

The scientific community describe by Kuhn are giving way to highly contextual and current paradigms and educational data is advancing more quickly than we know what do with it.This has created fragmentation that limits knowledge sharing and advance with different communities, different names, and different practices.It is not just that technology is driving the society but the society is driving the technology

We Hold These Truths to be Self Evident:Digital tools create vast quantities and categories of data: This datastorm is like nothing we have ever seen beforeQualitative Shift from institutional control of data, think personalized learning, people controlling their own learning. Bring your data with you like in health care.There is just so much more knowledge Th

The points here are twofold:There are parallels with other fields Even though we argue that education is different and has special properties, the movements in broad terms are similar and (very importantly) there are those who expect that education&apos;s transformation will parallel other fields. We are not arguing this, but there are parallels and this causes us to re-examine what about education is fundamentally different, unique.Education is both coming to this transformative stage late and with expectations to quickly catch up and so the time other fields have had to develop mature infrastructures was greater than education has leading to stresses in the process.Late to the game but that means that people expect EDS to be like finance and that is not goingThis is big and irrevesable.: OK, now we are in an age of big data and analytics and there is a lot of promise, a lot of potential, and a lot of need. At the same time, the field needs to develop a professional corps to deal with the new opportunity and challenges. It is not the

Now, this plugs my book. In doing so we can say that this book provides a broad argument for some of what to consider in this new paradigm, although there is more of a focus on K-12 than there could be. Specifically, the book:Compares education to other fields Explains using education data is necessary/difficultSynthesizes strands of education/organizational researchUsing design science and learning sciences framework

Traditionally called Institutional Research. Began as extension of analytic work and reporting many IHEs have been doing for years. FOCUSED ON INSTITUTIONAL NEEDSEarly warning systemsRecently, moving forward with new analyses including:Who gets accessHow they are admittedWhere they progress through the systemWhat is occurring in finance, fundraising, administration, grants management, etc.Moverment toward open standards (OAAI

Traditionally called Institutional Research. Began as extension of analytic work and reporting many IHEs have been doing for years. FOCUSED ON INSTITUTIONAL NEEDSEarly warning systemsRecently, moving forward with new analyses including:Who gets accessHow they are admittedWhere they progress through the systemWhat is occurring in finance, fundraising, administration, grants management, etc.Moverment toward open standards (OAAI

Largely from US with direct support No Child Left Behind (NCLB)Test focusedLots of talk about DDDMLimited by its data—lets just call it test driven decision making or even single test driven decision making.Ten years in an millions of dollars later we don’t have very much—But the notion of State Longitudinal Data Systems is big and includes more than test scoresNo conferences, no journals, some special issues but it is quite fragmentedlots of jobs and income for testing companiesSpecial issue of journalNo evidence of systemic validity, lots of evidence of negative consequenceIs getting a lot of attention because they will be using test scores to evaluated teacher performance

Largely from US with direct support No Child Left Behind (NCLB)Test focusedAccumulating research literature across many education communitiesLots of talk about DDDMLimited by its data—lets just call it test driven decision making or even single test driven decision making.Ten years in an millions of dollars later we don’t have very much—But the notion of State Longitudinal Data Systems is big and includes more than test scoresNo conferences, no journals, lots of jobsSpecial issue of journalNo evidence of systemic validity, lots of evidence of negative consequenceIs getting a lot of attention because they will be using test scores to evaluated teacher performance

Both Learning Analytics &amp; Educational Data Mining (EDM) emerged around same time (2005)Have similar roots in digital learning environments. Now share many aspects. Learning Analytics was more rooted in Includes use of data from LMSData Mining is cognitive tutors and videotamesBig focus at NSF right now (Big Data in Education)Example of broad convergence

How many of you think that these should merge?Lot of parallels with Learning Sciences and CSCL

Historically learners differences was cognitive (Hence the early Aptitude Treatment Interaction Work

These four areas are examples of the kinds of emerging data-intensive activities that work across the different parts of the data sciences

1. Rapid changing indicative of sociotechnical movementLike the steam engine and the printing press, lots of innovation across multiple sectors2. Boundary IssuesFor example, flipped classrooms stretch across systemic improvement and personalization, early warning systems stretch across EDM and Institutional Analytics.3. Disruption in evidentiary practicesAcross all four of these communities, there have also been questions about how to use different kinds of information that were previously not available; how to make high quality inferences using the different kinds of evidence in ways appropriate to the context4. Visualization, interpretation, and cultureAcross these four areas we also see the emergence of issues around visualization and interpretation of information. Visualization is usually the lead element in these discussions as different representational schemes. These include “dashboards” that rank and sort individuals and other similar tools used to make sense of the vast amounts of diverse information available in these four communities. 5. Ethics and privacyAcross all four of these areas are issues of ethics and privacy; how the collection and use of the information about learners and teachers can be done responsibly while also safeguarding the privacy of those whose information is captured.

Appreciate the Distinctive Character of Education DataIn important ways this field is like the other areas where data are used in other domains such as health care, finance, and industry. In some other ways, educational data has unique properties. These unique properties include:Human/social creation. Unlike most other fields that use data, much of educational data requires human manipulation, which increases the possibility of error and manipulation. Some have focused on cheating and gaming of the system in the area of tests, but this property is actually much more pervasive and affects areas like special education planning and school improvement plans as well as assessments. Measurement imprecision. Educational data can be rife with issues of precision, especially when assessments of student learning or systemic capabilities are used. Compared to blood pressure readings or financial transactions, educational assessments are noisy. They can be sensitive to student background, instructional techniques, circumstances of testing, and the likeComparability challenges. Comparisons across different areas of educational data can be sometimes impacted by contact variation. For example, different schools are often compared for many different kinds of analyses. However, programmatic variation often occurs from school to school and those programmatic differences may not always be apparent in the data streams. Fragmentation. The world of educational data is fragmented. Many different organizations hold parts of educational information and there are still incomplete and partially adopted technical standards which impacts the ability to link some data without specific extra work. There are a number of efforts to create interoperable data standards. While progress has been made in these areas the road forward will be difficult as the governance of educational data is highly decentralized owing to the US Constitution’s delegation of authority for education to states and across the states there are many different approaches to state-district interactions and almost 20,000 district and charter providers.

The message here is that these six fields (not counting computer science) are all influential in the EDS, but computer science is more so because it is generative with the development of different innovations that will end up rippling through other fields and then to EDS. Roy Pea was behind this thinking.

The concept here is that the same artifacts can serve multiple purposes. This ties back nicely to the paradigm shift because historically data were developed for single purposes and often externally. Now, data is generated from many places and being used for many purposes.

This is really the sociotechnical thesis that argues that the artifacts – any artifacts – are not neutral. They are both adopted and adapted. They shape contexts and so the implications of them are far beyond their singular instrumental use. For example, think of NCLB testing. These are more than simply tests that measured student achievement. They cast a huge shadow over following years and were exploited by conservatives to help break the public’s confidence in public schools

The Gates Data Quality Campaign likes to use the phrase data should be like a flashlight not a hammer. I go further to say it is like lenses, imperfect, but generally better than nothing. Educational data is rarely truth, but often an indication of something to pay attention to.

Let me summarize the four main points in our paper and in this talk.Paradigm shift in conception of data. It is a sociotechnological one.I will then argue that this movement involves four distinct fields and that together these fields might be most productive if merged into one broader notion of EDSWe then discuss five common features that unite the currently disparate EDS communities WE then

Phil is with ed info connections in DCI am with IUMJ Bishop is the director of CIELT for the university system of Maryland

13.
Systemic/Instructional Improvement
“In many ways, the practice of data use is out ahead of research.
Policy and interventions to promote data use far outstrip research
studying the process, context, and consequences of these efforts.
But the fact that there is so much energy promoting data use and
so many districts and schools that are embarking on data use
initiatives means that conditions are ripe for systematic, empirical
study.”
Coburn, Cynthia E., and Erica O. Turner. "Research
on data use: A framework and analysis."
Measurement: Interdisciplinary Research &
Perspective 9.4 (2011): 173-206.
13