Schools, Science and Education

A few years back I went for a pizza with an old friend. We shared a pretty large pizza but somehow ended up with just one slice left between the two of us which we both desperately wanted. Bearing in mind that we would both happily lie, trick or outright fight each other for the last slice, our ensuing discussion about how to apportion it ended in a stalemate, with neither of us agreeing on a compromise. At this point, my friend suggested a solution: I would cut the slice in half, and he would choose which half to take.

Brilliant. It was in my interests to cut it as equally as possible, guaranteeing that we would both get those last few mouthfuls of now-cold pizza. Because I didn’t know which half I would get, the only logical path was to cut it equally.

***

Without knowing it, my friend had stumbled across an incredibly important theory from political philosophy: John Rawls’ Veil of Ignorance. The Veil is a way of distributing the goods and resources of society equally. Essentially, you decide to structure society – its rules and its norms – before you know which role you will take within that society. So you wouldn’t set up a society where 20% of citizens are slaves, because you don’t know if you are going to be a slave or free. You wouldn’t set it up under patriarchal lines, because you don’t know if you will be male or female. You would include access for the disabled in infrastructure, equitable toilets in public spaces and social support for those struggling with ill health as who knows – perhaps you will be disabled, will want equitable access to a toilet or will have poor physical or mental health. Not knowing if you were to be gay or straight, would you not legalise gay marriage? This is how Rawls argued we should set the rules and mores of our society.

It’s a great thought experiment for a couple of reasons. First, because it forces you to think of the Other. Normally whenever thinking about such charged topics it’s difficult to divorce our private personal experiences and histories from our public opinions about the structure of society and its laws. Middle class, working class, black, white, gay, straight, atheist, Jew or Christian – it forces you to anticipate everyone’s needs. You don’t know who you are going to be, so you have to think about what it must be like to be absolutely anyone else. If I were to be that person, what would I want? What would I need?

The second reason I think it’s important is that it forces you into a radical re-imagining of what things could be like. If you started from scratch, if you hit the “delete” button on everything you had: every rule, every policy, every law and every structure – what would the world look like?

***

You can play this game with smaller entities than entire countries too. A school would be a prime candidate. Most schools I know have over the years built up a huge amount of clutter: “new” policies built on the remains of old policies, departmental inconsistencies, cultures that are a messy patchwork of hearsay, experience from other schools and barely remembered CPD. School life and culture is an often incoherent amalgam of historical detritus: the debris left in the wake of any large organisation strapped for time and cash.

Such an environment is ripe for the question to be asked: what would it look like if we could start again? What if we could de-clutter the detritus and start afresh? The Veil is perfect for this.

The Veil is helpful for another reason too: in my admittedly limited experience I’ve found that different people in a school don’t really know what their colleagues do most of the time, and therefore cannot understand or anticipate what they need in order to do their job effectively. Teachers might not understand the needs of the head, the head might not understand the needs of a TA, a TA might not understand the needs of the School Business Manager who in turn may not understand the needs of the parents. Students, parents, governors, heads of department, site staff: all fit into this mess of conflicting needs and agendas. Each of these people fulfills a vital role in a school, and the needs of each must be considered when designing policy.

***

I’ve been reading a lot of articles about school use of data recently. I don’t consider myself an expert. I know some people who really are, and I’m not one of them. I’ve asked a lot of questions and I’ve read a lot of articles on the topic but I’m still way behind. I do think that data would be an interesting case study for our Veil thought experiment, and as many schools are thinking about their data policies the Veil could prove a useful theoretical framework through which to assess and adapt school data policies. The framework would boil down to two simple questions:

In terms of data, what are the needs of the different agents in a school environment?

Starting from scratch, how could we design a system that caters to all their needs?

Below are my tentative steps to addressing these questions. I’m not going to answer them directly, but I am going to point to a number of areas that I think are worth considering. With Ofsted announcing that they will no longer look at internal data, there has never been a better time to radically re-imagine your school’s data policy. I haven’t linked to everything that I have read on the topic, but I have collected a number of articles here.

Trust

Do you trust your staff? If you’re the one writing the policy, do you trust what your staff have to say about your students? Maybe, maybe not. But sure as hell if you were a teacher you would want to be trusted. And that’s the point of the exercise – you don’t know which role you will take. So you have to build a system predicated on trusting your staff, and providing them with the means to make judgments which they are confident in.

2. Who needs what?

People need different things from the data. If you decide that your deputy head needs detailed sub-levels colour coded and subtracted from target grades that’s fine (though I wouldn’t recommend it), but it may not be what the parents need. If you were a parent of a child at your school, what do you actually need to know from the school? Different roles need different things.

3. Workload

I managed to get to point 3 before talking about workload. As a Head or Deputy with only a handful of classes, it might feel reasonable to ask for three, four or five data drops a year. But if you were to be a frontline teacher actually implementing this policy with 14 classes…maybe not so much. Equally, frontline staff need to appreciate the need for strategic planning and preparation. If you were going to be a deputy head, you would want to have some kind of information about the performance of students in their various subjects. A compromise – from behind the security of the veil – must be made.

4. Won’t somebody please think of the children??

Flippant, but apt. Remember, you don’t know what role you are going to have. You might be a student who really struggles at school and, four times a year like clockwork, gets told that they are under target. Or that they are on target, but their target is a fail. Not so nice. I know I wouldn’t want to be that student.

5. Accountability

All roles in a school need, at some point, to be accountable for their performance. I don’t think many would disagree with that. I want to know that somebody is keeping an eye on me, and will haul me up if I do something wrong or stop performing. But one of the roles in a school is “teacher with tough classes” or “teacher who ends up with bottom sets” or “teacher who has inherited a class who know bugger all.” These, and any number of other types of teacher must surely invalidate the use of data as a performance metric. It’s just too complicated. There are too many variables – too many things outside of the teacher’s control.Even more so, I also might not be a teacher but the person looking at the data, at which point I want to know that it is uncorrupted and gives me information about student performance, not about how teachers are playing it safe so as not to provoke a passive aggressive email from their line manager or a request from on high to put in more interventions for students who routinely mess around in their classes. If accountability is desirable, but based on data will corrupt and become self-defeating, maybe find a different route to accountability?

6. Infernal inferences

Data is just a string of numbers: a test score, an average test score, a residual or whatever. You are using them to infer something about something. So you might use a low mock exam result to infer that a student did not do enough work for that mock. But what if the student’s dog died the day before? Or they worked, but their methods were ineffective? Or their teacher hadn’t adequately prepared them or this or that or any number of things. It can be a dangerous games making inferences, and none of us want to be in a role where we are making bad inferences or, perhaps worse, to be in a role where bad inferences are being made about us.

7. What’s next?

Similar to inferences is the role of action. What is going to be done about this data? Do you want to be a teacher who has to spend hours each year entering data from which nothing will actually be done? Doubt it.

8. Aggregation is the friend of reliability…

A phrase borrowed from Becky Allen, it’s an encouragement to think about how many data points your inferences are based on. In short, inferences from one test are not that reliable. Two tests: more reliable. More students, more tests = more reliable. As Becky points out, you don’t need an actual test for this. Any time a teacher looks at a student’s homework, or listens to their answers in class they are taking mental data on that student. All of that counts.

9. …but more crap tests will just give you crap data

So this is where curriculum and assessment comes in. What are your tests testing? Are they doing it well? In the past we used tests which weren’t well tied to what had been taught, had confusing questions or any number of other technical flaws. You don’t want to be a student sitting a test which doesn’t test what you learned, and you don’t want to be the teacher that wastes their, and their students time, like that. So if you are going to spend time on something, whatever your role, maybe just say “right no data drops this year, but everyone is going to give 15 hours of their time across the year to making sure the assessments they are using make sense.”

10. You’re all bright and sparkly now, but remember you could be me

At a conference a little while ago I saw a head of department presenting their incredibly complex and detailed RAG sheet for student mocks. They were all bright and sparkly and talking about all the wonderful things that they had learned from entering all that data and analysing the ace colour palette. Look, if you think the role you will take is that of a workaholic head of department who will martyr their Sundays on the altar of presenting shiny things at conferences and getting #pedagoo likes on Twitter that’s fine. But remember: you could end up being me. And I would rather shave my eyeballs with a rusty switchblade than fill out a rag sheet. Just something to think about.

***

That’s probably it for now. If anyone knows of some brilliant practice on this front or really interesting reading please do let me know.

Like this:

Adam

16 thoughts on “Data’s veil of ignorance”

We pretty much went through this process – trying to work out, from a teaching perspective (but also bearing in mind students, school leaders, parents etc) what data we would be most interested in about students and how we would collect it.

Being insanely ambitious, we designed an idea of what we wanted: a huge transferable curriculum, spanning multiple subjects, with sound content, the opportunity for students to explore beyond the confines of their course and for everyone to see what they’ve covered. Which was basically an online textbook for students to tick off as the6 work through.

Then we added thousands upon thousands of quality questions of different types, each one linked to one (or more) concepts. That is the testing taken care of. We then thought teachers should be able to custom-create their own assessments by searching by difficulty/concept or whatever.

Then we added detailed feedback for each question. Mammoth task, but as teachers we can largely anticipate what students often get wrong. How do we mark essays or written responses? Students need to self-assess, so we added that. What if they lie? Teachers need to review/override marks/be able to offer feedback, so we added that too.

As if this isn’t ambitious enough, we also designed it so that all the data would be collected automatically. No more data dumps, ever.

It’s a very thought-provoking read and I’m grateful to you for writing it. I agree with loads of what you say.

My issue is that I don’t think the ‘veil of ignorance’ is a good basis for making decisions about school data, even though it might be for a just society. Your argument is based on an analogy between society and schools, which I don’t think is valid. This is because society exists (arguably) to enable all its members to flourish and we make the assumption that they are able to make autonomous decisions about what they want from life, hence the power of the ‘veil’ as a method in putting equal weight on the needs of all those members.

However, this does not apply to schools, which are a tool of society, set up for the specific purpose of educating young people (a social good). Therefore we make a different assumption: that education is good for students, even if they think differently, and that we almost certainly know better than they do what that education should comprise.

As a result, we cannot design a system for school data which assumes that what everyone wants (or even needs) has equal weighting to share out the happiness in some way (even if that were possible). On the contrary, systems involving data, and all others, must be designed to ensure they do the best job possible of helping schools to achieve their core goal of educating (although of course other things have to be taken into account, such as workload). So I would argue that the opinion of an expert (e.g. a headteacher) ought to carry more weight than what a student or parent might want (if not, we would have legitimate questions about that headteacher’s competence).

Now I would argue that if we apply this criterion for judging data systems (whether or not they enable schools to do the best job possible of educating students), we would still sweep away an awful lot of the accumulated detritus which you write about so eloquently. I might even come to the same conclusions as you about what works, but not quite for the same reasons.

That’s a really interesting and thoughtful comment – thank you. I think what I am going to write is academic, because I do agree with you and I am sure that if we both went into a separate room to write a policy we would be 99% in the same place.

What I might say to you in response to the applicability of the Veil in particular regarding parents, it was not my intent to say that they should be dictating what kind of data we are collecting, but when it is reported back to them only the type of data that we as a parent would want to see should be reported. So when you see these crazy report cards full of numbers and statistics that are thoroughly unintelligible to the lay person, you have not properly anticipated the actual needs of the parent who is left bewildered and confused.

In terms of the veil’s applicability to a school, I would argue that when you are abstracting yourself from your role to behind the veil, all future roles agree that education is a social good. So that would mean that within the role of parent would be things like concern over safeguarding and quality of education etc, but I would also include in that the assumption that the experts know best. Introducing a role of “parent who just hates the school and thinks that they know best” then yes, the model collapses. But it would also collapse when applied to political systems if one of the roles was “sociopath who wants to be able to murder people and not go to prison.” So I’m not sure that it is really a valid distinction of the model as it exists in the “world of ideas.” Does that make sense?

Really enjoyed this, thanks! Using the veil as a way of examining data is a really interesting point. But (and I haven’t sorted this out fully in my head), the reason we examine data on performance is because there are inherent inequalities that need explication. Therefore, is a thought experiment that explicitly erases inequalities as the start point an appropriate tool? The fact we need data in the first place implicitly acknowledges unequal power relationships and a struggle for resources. If indeed valid, the veil of ignorance experiment might be far more appropriately deployed further upstream from the point we look at data. However, the fact there is an unequal power relationship implied in the collection of data would mean the experiment would be performed under the the auspices of the powerful (HTs, DfE etc), thus subjecting all considerations to their own viewpoint and cognitive biases. That isn’t their fault – no one can look through different eyes and inhabit another person’s experience. I can imagine it would be very fertile grounds for post-hoc rationalisation.

In terms of whether it is logically valid in theory or in the real world?
As I say, I haven’t sorted it all out in my own head – I’m thinking out loud so happy to be wrong/encouraged in another direction.
I assume you mean in the real world though. The example you mention of the rag sheet is one such example. The person doing that is likely to have power enough to rubber stamp such an exercise or at the very least profoundly influence it, but because they’re the kind of person that enjoys rag sheet creation on a Sunday weekend, they are less likely to understand why anyone wouldn’t. Therefore, their decisions about responding to their perceived understanding of ‘the other’ will always be filtered through their own experiences, values and preferences – the rag sheets are worthwhile so deserve to stay. This will be influenced by their relative position of power, which is what gives them the ability to start an application of the experiment in the first place – an uncomfortably circular conversation where the HoD confronts his/her own biases by rationalising with his/her own biases.

Ah right yes now I think I follow. This is one of the classic criticisms of Rawls’ theory – that it is actually impossible to genuinely abstract yourself (the second being that it is perhaps undesirable). You could well be right, but I can’t think of a better way to help such people try to put themselves in others’ shoes!

Search

Search for:

Text Widget

This is a text widget, which allows you to add text or HTML to your sidebar. You can use them to display text, links, images, HTML, or a combination of these. Edit them in the Widget section of the Customizer.