the venomous data bore

Robert Peal has posted a series of responses to critics of his book Progressively Worse here. The second is on ‘data and dichotomies’. In this post I want to comment on some of the things he says about data and evidence.

when ‘evidence doesn’t work’

Robert* refers back to a previous post entitled ‘When evidence doesn’t work’ summarising several sessions at the ResearchED conference held at Dulwich College last year. He rightly draws attention to the problem of hard-to-measure outcomes, and to which outcomes we decide to measure in the first place. But he appears to conclude that there are some things – ideology, morality, values – that are self-evidently good or bad and that are outside the remit of evidence.

In his response to critics, Robert claims that one reason ‘evidence doesn’t work’ is because “some of the key debates in education are based on value judgements, not efficacy.” This is certainly true – and those key debates have resulted in a massive waste of resources in education over the past 140 years. There’s been little consensus on what long-term outcomes people want from the education system, what short-term outcomes they want, what pedagogies are effective and how effectiveness can be assessed. If a decision as to whether Shakespeare ‘should’ be studied at GCSE is based on value judgements it’s hardly surprising it’s been the subject of heated debate for decades. Robert’s conclusion appears to be that heated debate about value judgements is inevitable because values aren’t things that lend themselves to being treated as evidence. I disagree.

data

I think he draws this conclusion because his view of data is rather limited. Data don’t just consist of ‘things we can easily measure’ like exam results (Robert’s second reason why ‘evidence doesn’t work’). They don’t have to involve measuring things at all; qualitative data can be very informative. Let’s take the benefits of studying Shakespeare in school. Robert asks “Can an RTC tell us, for example, whether secondary school pupils benefit from studying Shakespeare?” If it was carefully controlled it could, though we would have to tackle the question of what outcomes to measure. But randomised controlled trials are only one of many methods for gathering data. Collecting qualitative data from a representative sample of the population about the impact studying Shakespeare had had on their lives could give some insights, not only into whether Shakespeare should be studied in school, but how his work should be studied. And whether people should have the opportunity to undertake some formal study of Shakespeare in later life if they wanted to. People might appreciate actually being asked.

venomous data bore*

venomous data bore Buprestis octoguttata§

opinion

I don’t know whether Robert sees me as what he refers to as a ‘data bore’, but if he does I accept the epithet as a badge of honour. For the record however, not only have I never let a skinny latte pass my lips, but the word ‘nuanced’ has never done so either (not in public, at least). Nor do I have a “lofty distain for anything so naïve as ‘having an opinion’”.

I’m more than happy for people to have opinions and to express them and for them to be taken into account when education policy is being devised. But not all opinions are equal. They can vary between professional, expert opinion derived from a thorough theoretical knowledge and familiarity with a particular research literature, through well-informed personal opinion, to someone simply liking or not liking something but not having a clue why. I would not want to receive medical treatment based on a vox pop carried out in my doctor’s waiting room, nor do I want a public sector service to be designed on a similar basis. If it is, then the people who voice their opinions most loudly are likely to get what they want, leaving the rest of us, ‘data bores’ included, to work on the damage limitation.

rationality and values

Robert appears to have a deep suspicion of rationality. He says “rational man believes that they can make their way in the world without recourse to the murky business of ideology and morality, or to use a more contemporary term, ‘values’.” He also says it was ‘terrific’ to hear Sam Freedman expound the findings of Jonathan Haidt and Daniel Kahnemann “about the dominance of the subconscious, emotional part of our minds, over the logical, conscious part.” He could add Antonio Damasio to that list. There’s little doubt that our judgement and decision-making is dominated by the subconscious emotional part of our minds. That doesn’t mean it’s a good thing.

Ideology, morality and values can inspire people to do great things, and rationality can inflict appalling damage, but it’s not always like that. Every significant step that’s been ever been taken towards reducing infant mortality, maternal mortality, disease, famine, poverty and conflict and every technological advance ever made has involved people using the ‘logical conscious part’ of their minds as well as, or instead of, the ‘subconscious emotional part’. Those steps have sometimes involved a lifetime’s painstaking work in the teeth of bitter opposition. In contrast, many of the victims of ideology, morality and values lie buried where they fell on the world’s battlefields.

Robert’s last point about data is that they are “simply not able to ‘speak for themselves’. Its voice is always mediated by human judgement.” That’s not quite the impression given on page 4 of his book when referring to a list of statistics he felt showed there was a fundamental problem in British education. In the case of these statistics, ‘the bare figures are hard to ignore’.

Robert is quite right that the voice of the data is always mediated by human judgement, but we have devised ways of interpreting the data that make them less susceptible to bias. The data are perfectly capable of speaking for themselves, if we know how to listen to them. Clearly the researcher, like the historian, suffers from selection bias, but some fields of discourse, unlike history it seems, have developed robust methodologies to address that. The biggest problem faced by the data is that they can’t get a word in edgeways because of all the opinion being voiced.

endnote

According to this tweet from Civitas…

civitas venom

Robert says he has responded to criticism in blogs by Tim Taylor, Guy Woolnough and myself. I’m doubtless biased, but the comment most closely resembling ‘venom’ that I could find was actually in a scurrilous tweet from Debra Kidd, shown in Robert’s third response to his critics. Debra, shockingly for a teacher, uses a four-letter-word to describe Robert’s description of state schools as ‘a persistent source of national embarrassment’. She calls it ‘tosh’. If Civitas thinks that’s venom, it clearly has little experience of academia, politics or the playground. Rather worrying on all counts, if it’s a think tank playing a significant role in education reform.

* I felt we should be on first name terms now we’ve had a one-to-one conversation about statistics.

§ Image courtesy Christian Fischer from Britannica Kids.

It’s not really a venomous data bore, it’s a Metallic wood-boring beetle. It’s not really metallic either, it just looks like it. Nor does the beetle bore wood, its larvae do. Words can be so misleading.

a tale of two Blobs

The think-tank Civitas has just published a 53-page pamphlet written by Toby Young and entitled ‘Prisoners of The Blob’. ‘The Blob’ for the uninitiated, is the name applied by the UK’s Secretary of State for Education, Michael Gove, to ‘leaders of the teaching unions, local authority officials, academic experts and university education departments’ described by Young as ‘opponents of educational reform’. The name’s not original. Young says it was coined by William J Bennett, a former US Education Secretary; it was also used by Chris Woodhead, first Chief Inspector of Ofsted in his book Class War.

It’s difficult to tell whether ‘The Blob’ is actually an amorphous fog-like mass whose members embrace an identical approach to education as Young claims, or whether such a diverse range of people espouse such a diverse range of views that it’s difficult for people who would like life to be nice and straightforward to understand all the differences.

Young says;

They all believe that skills like ‘problem-solving’ and ‘critical thinking’ are more important than subject knowledge; that education should be ‘child-centred’ rather than ‘didactic’ or ‘teacher-led’; that ‘group work’ and ‘independent learning’ are superior to ‘direct instruction’; that the way to interest children in a subject is to make it ‘relevant’; that ‘rote-learning’ and ‘regurgitating facts’ is bad, along with discipline, hierarchy, routine and anything else that involves treating the teacher as an authority figure. The list goes on.” (p.3)

It’s obvious that this is a literary device rather than a scientific analysis, but that’s what bothers me about it.

Initially, I had some sympathy with the advocates of ‘educational reform’. The national curriculum had a distinctly woolly appearance in places, enforced group-work and being required to imagine how historical figures must have felt drove my children to distraction, and the approach to behaviour management at their school seemed incoherent. So when I started to come across references to educational reform based on evidence, the importance of knowledge and skills being domain-specific, I was relieved. When I found that applying findings from cognitive science to education was being advocated, I got quite excited.

My excitement was short-lived. I had imagined that a community of researchers had been busily applying cognitive science findings to education, that the literatures on learning and expertise were being thoroughly mined and that an evidence-based route-map was beginning to emerge. Instead, I kept finding references to the same small group of people.

Most fields of discourse are dominated by a few individuals. Usually they are researchers responsible for significant findings or major theories. A new or specialist field might be dominated by only two or three people. The difference here is that education straddles many different fields of discourse (biology, psychology sociology, philosophy and politics, plus a range of subject areas) so I found it a bit odd that the same handful of names kept cropping up. I would have expected a major reform of the education system to have had a wider evidence base.

Evaluating the evidence

And then there was the evidence itself. I might be looking in the wrong place, but so far, although I’ve found a few references, I’ve uncovered no attempts by proponents of educational reform to evaluate the evidence they cite.

A major flaw in human thinking is confirmation bias. To represent a particular set of ideas, we develop a mental schema. Every time we encounter the same set of ideas, the neural network that carries the schema is activated. The more it’s activated, the more readily it’s activated in future. This means that any configuration of ideas that contradicts a pre-existing schema, has, almost literally, to swim against the electromagnetic tide. It’s going to take a good few reiterations of the new idea set before a strongly embedded pre-existing schema is likely to be overridden by a new one. Consequently we tend to favour evidence that confirms our existing views, and find it difficult to see things in a different way.

The best way we’ve found to counteract confirmation bias in the way we evaluate evidence is through hypothesis testing. Essentially you come up with a hypothesis and then try to disprove it. If you can’t, it doesn’t mean your hypothesis is right, it just means you can’t yet rule it out. Hypothesis testing as such is mainly used in the sciences, but the same principle underlies formal debating, the adversarial approach in courts of law, and having an opposition to government in parliament. The last two examples are often viewed as needlessly combative, when actually their job is to spot flaws in what other people are saying. How well they do that job is another matter.

It’s impossible to tell at first glance whether a small number of researchers have made a breakthrough in education theory, or whether their work is simply being cited to affirm a set of beliefs. My suspicion that it might be the latter was strengthened when I checked out the evidence.

The evidence

John Hattie conducted a meta-anlaysis of over 800 studies of student achievement. My immediate thought when I came across his work was of the well-documented problems associated with meta-analyses. Hattie does discuss these, but I’m not convinced he disposed of one key issue; the garbage-in-garbage-out problem. A major difficulty with meta-analyses is ensuring that all the studies involved use the same definitions for the constructs they are measuring; and I couldn’t find a discussion of what Hattie (or other researchers) mean by ‘achievement’. I assume that Hattie uses test scores as a proxy measure of achievement. This is fine if you think the job of schools is to ensure that children learn what somebody has decided they should learn. But that assumption poses problems. One is who determines what students should learn. Another is what happens to students who, for whatever reason, can’t learn at the same rate as the majority. And a third is how the achievement measured in Hattie’s study maps on to achievement in later life. What’s noticeable about the biographies of many ‘great thinkers’ – Darwin and Einstein are prominent examples – is how many of them didn’t do very well in school. It doesn’t follow that Hattie is wrong – Darwin and Einstein might have been even greater thinkers if their schools had adopted his recommendations – but it’s an outcome Hattie doesn’t appear to address.

Siegfreid Engelmann and Wesley C Becker developed a system called Direct Instruction System for Teaching Arithmetic and Reading (DISTAR) that was shown to be effective in Project Follow-Through – a evaluation of a number of educational approaches in the US education system over a 30 year period starting in the 1960s. There’s little doubt that Direct Instruction is more effective than many other systems at raising academic achievement and self-esteem. The problem is, again, who decides what students learn, what happens to students who don’t benefit as much as others, and what’s meant by ‘achievement’.

ED Hirsch developed the Core Knowledge sequence – essentially an off-the-shelf curriculum that’s been adapted for the UK and is available from Civitas. The US Core Knowledge sequence has a pretty obvious underlying rationale even if some might question its stance on some points. The same can’t be said of the UK version. Compare, for example, the content of US Grade 1 History and Geography with that of the UK version for Year 1. The US version includes Early People and Civilisations and the History of World Religion – all important for understanding how human geography and cultures have developed over time. The UK version focuses on British Pre-history and History (with an emphasis on the importance of literacy) followed by Kings and Queens, Prime ministers then Symbols and figures – namely the Union Jack, Buckingham Palace, 10 Downing Street and the Houses of Parliament – despite the fact that few children in Y1 are likely to understand how or why these people or symbols came to be important. Although the strands of world history and British history are broadly chronological, Y4s study Ancient Rome alongside the Stuarts, and Y6s the American Civil War potentially before the Industrial Revolution.

Daniel Willingham is a cognitive psychologist and the author of Why don’t students like school? A cognitive scientist answers questions about how the mind works and what it means for the classroom and When can you trust the experts? How to tell good science from bad in education. He also writes for a column in American Educator magazine. I found Willingham informative on cognitive psychology. However, I felt his view of education was a rather narrow one. There’s nothing wrong with applying cognitive psychology to how teachers teach the curriculum in schools – it’s just that learning and education involve considerably more than that.

Kirschner, Sweller and Clark have written several papers about the limitations of working memory and its implications for education. In my view, their analysis has three key weaknesses; they arbitrarily lump together a range of education methods as if they were essentially the same, they base their theory on an outdated and incomplete model of memory, and they conclude that only one teaching approach is effective – explicit, direct instruction – ignoring the fact that knowledge comes in different forms.

Conclusions

I agree with some of the points made by the reformers:
• I agree with the idea of evidence-based education – the more evidence the better, in my view.
• I have no problem with children being taught knowledge. I don’t subscribe to a constructivist view of education – in the sense that we each develop a unique understanding of the world and everybody’s worldview is as valid as everybody else’s – although cognitive science has shown that everybody’s construction of knowledge is unique. We know that some knowledge is more valid and/or more reliable than other knowledge and we’ve developed some quite sophisticated ways of figuring out what’s more certain and what’s less certain.
• The application of findings from cognitive science to education is long overdue.
• I have no problem with direct instruction (as distinct from Direct Instruction) per se.

However, some of what I read gave me cause for concern:
• The evidence-base presented by the reformers is limited and parts of it are weak and flawed. It’s vital to evaluate evidence, not just to cite evidence that at face-value appears to support what you already think. And a body of evidence isn’t a unitary thing; some parts of it can be sound whilst other parts are distinctly dodgy. It’s important to be able to sift through it and weigh up the pros and cons. Ignoring contradictory evidence can be catastrophic.
• Knowledge, likewise, isn’t a unitary thing; it can vary in terms of validity and reliability.
• The evidence from cognitive science also needs to be evaluated. It isn’t OK to assume that just because cognitive scientists say something it must be right; cognitive scientists certainly don’t do that. Being able to evaluate cognitive science might entail learning a fair bit about cognitive science first.
• Direct instruction, like any other educational method, is appropriate for acquiring some types of knowledge. It isn’t appropriate for acquiring all types of knowledge. The problem with approaches such as discovery learning and child-led learning is not that there’s anything inherently wrong with the approaches themselves, but that they’re not suitable for acquiring all types of knowledge.

What has struck me most forcibly about my exploration of the evidence cited by the education reformers is that, although I agree with some of the reformers’ reservations about what’s been termed ‘minimal instruction’ approaches to education, the reformers appear to be ignoring their own advice. They don’t have extensive knowledge of the relevant subject areas, they don’t evaluate the relevant evidence, and the direct instruction framework they are advocating – certainly the one Civitas is advocating – doesn’t appear to have a structure derived from the relevant knowledge domains.

Rather than a rational, evidence-based approach to education, the ‘educational reform’ movement has all the hallmarks of a belief system that’s using evidence selectively to support its cause; and that’s what worries me. This new Blob is beginning to look suspiciously like the old one.