Validity and Values: The debate about education research.

“It seems to me what is called for is an exquisite balance between two conflicting needs: the most skeptical scrutiny of all hypotheses that are served up to us and at the same time a great openness to new ideas … If you are only skeptical, then no new ideas make it through to you … On the other hand, if you are open to the point of gullibility, … then you cannot distinguish the useful ideas from the worthless ones.”

Carl Sagan in “The Burden of Skepticism”

Much of the debate regarding the role of research evidence in education appears to involve a series of misconceptions about what evidence can and – perhaps more importantly – cannot tell us.

The nature of empirical claims
Education has long been vulnerable to untested – or simply false – empirical claims. These have frequently been accepted and propagated (through CPD and teacher training) without question. For example:

• “Students must drink water in lessons or else their brains will shrink and they won’t be able to concentrate.”
• “Students retain 75% of what they do but only 10% of what they read.”
• “Students who are right-brain dominant learn differently to students who are left-brain dominant.”

In some cases these claims have lacked validity – e.g. they directly contradict some pretty basic biology (brain shrinkage). In other cases the claims have been demonstrated as empirically false (e.g. the pyramid of learning) but most teachers appear unaware of the evidence.

In many cases, those sceptical of education research aren’t against this use of evidence. Weeding out such spurious claims and nonsense appears a fairly uncontroversial use of ‘evidence in education’.

However, in addition to challenging spurious claims, evidence-based research can help inform teachers about strategies and techniques which they may find useful in the classroom – or ones might help groups of students within a school population.

• “Direct instruction leads to greater retention of abstract scientific concepts than inquiry-based learning.”
• “Students whose teacher has more knowledge of the misconceptions related to their subject make better progress than students whose teacher does not.”
• “Pupils with below-average reading scores catch up with their peers more quickly with intensive reading intervention A rather than intervention B.”

These are the types of question that are open to evidence-based research. That’s the honest purpose of evidence – to test empirical claims.

The nature of value statements

“Friendship is unnecessary, like philosophy, like art. . . . It has no survival value; rather it is one of those things that gives value to survival.”

C. S. Lewis in “The Four Loves”

However, the debate within education is much wider than simply empirical claims about how kids learn and what sort of strategies might help particular groups of students in schools. Where evidence-based research stops is where our values start. Value statements tend to phrased differently: They tend to suggest something ‘should’ be the case rather than ‘is’.

Some value statements:

• “Children should play more chess in schools.”
• “It’s important that pupils read Shakespeare.”
• “We should challenge racism, sexism and homophobia.”

No amount of science can tell you what your values should be. These values cannot be contested or compared by any amount of research. They are personal statements of social aspiration rather than empirical claims. To fight for your values you must argue and persuade other people why they are right: write a polemic article for a newspaper, lobby your MP, argue with people at conferences …

Attempts to prop up such value statements by cherry picking or manufacturing ‘evidence’ is at best naïve and at worst dishonest. We can’t use ‘science’ to decide our values. That’s not what science is for. Science is a tool for helping us understand ‘what is’ (though imperfectly) – not ‘what should be’.

Where values and evidence interact
One reason many education debates appear to get muddled is because the value sets of individuals conflict with the outcomes measured in an empirical study. Thus an empirical question …

“Direct instruction leads to greater retention of abstract scientific concepts than inquiry-based learning.”

… pre-supposes that ‘greater retention of abstract scientific concepts’ is a ‘desirable thing’ for children to develop through school. This is a problem that science can’t help with. For people whose values hold that such an outcome is desirable – such research evidence has significance. For people whose values oppose this outcome – such research evidence is irrelevant.

Teachers often hold different value-sets when it comes to ‘what is important’ about education. Where people are suspicious of evidence-informed practice it is quite frequently because they disagree with the values embedded in the outcome measures used in the research – and the feeling that ‘evidence’ is being used to shut down the discussion about ‘what’s important’ in the first place.

Some ways forward: Are we arguing about validity or values?
There are different sorts of arguments about education research – and the frequent conflation of validity with values serves to create more heat than light in discussions.

One thing that helps clarify debates about education is to be clear what it is you are disagreeing with. It helps when those values are articulated clearly and early in a discussion. An example of this is Coe et al (2014) in their recent review ‘What makes great teaching?’

“We define effective teaching as that which leads to improved student achievement using outcomes that matter to their future success. Defining effective teaching is not easy. The research keeps coming back to this critical point: student progress is the yardstick by which teacher quality should be assessed. Ultimately, for a judgement about whether teaching is effective, to be seen as trustworthy, it must be checked against the progress being made by students.”

By clearly defining ‘great teaching’ in this way, Coe et al make clear the value-set driving the research – ‘progress made by students is desirable’.

There are fundamentally different ways to disagree with this:

You could argue that the notion of ‘progress made by students’ shouldn’t be the defining characteristic of great teaching. This is a rejection of the values of the research. Disagreeing in this way moves the discussion out of the realm of ‘science’ or ‘evidence’; into the political arena where you will have to persuade others of the virtue of your alternative values.

You could argue that the research methods used did not measure progress. This is an argument about the validity of the research, but doesn’t disagree with the values. Disagreeing in this way keeps the discussion within the realm of ‘science’ and ‘evidence’; you will need to explain why the measurements used were not valid – and (better still) suggest more valid ways to make such measurements.

Some ways forward: Value statements and empirical claims don’t mix
Where individuals are clearly arguing about values, the interjection of ‘evidence’ is unhelpful and feels like one party is trying to shut down the debate. This is likely the reason that some teachers have a problem with the idea of evidence-informed practice. There’s the feeling that the whole discussion about ‘what is important’ has been side-lined by the shift of focus to evidence claims.

However, when arguing about values it’s relatively easy to slip into empirical statements. For example:

“I don’t believe we should judge student success in school in terms of exam results. All this emphasis on exams just makes children anxious and miserable.”

The first part of this statement requires no empirical evidence – it’s simply an expression of values. You may have to argue why exam results shouldn’t be a priority, or what should be a priority in their place, but ‘evidence’ isn’t the issue. The second part of this statement is making an empirical claim – you should be prepared to support that statement with some sort of evidence about the effects of exams on student well-being.

Some ways forward: Are there common professional values?
What’s badly needed within teaching is some identification and articulation of genuinely common values. Here, arguably, is where teaching is very different from other professions – like medicine. For medical doctors the ‘purpose of medicine’ has been more or less agreed within the profession, but can we find a common ‘purpose of education’?

The reason, perhaps, that education remains such a political football isn’t the lack of empirical methods underpinning our instructional techniques – I suggest – but that the lack of commonly articulated values which allows/encourages politicians to dictate those values to us. For education to move forward – and for teachers to gain greater recognition as a profession – there is a need to identify common ground in terms of values.

Perhaps there aren’t any, and like ‘British values’, attempts to articulate them will only make our divisions more sharply defined! We’re a diverse profession – individuals from a wide variety of social backgrounds and academic disciplines – but it’s hard to believe there isn’t something, even something quite small and simple, which unites us all?