education reform: evaluating the evidence

Over the last few weeks I’ve found myself moving from being broadly sympathetic to educational ‘reform’ to being quite critical of it. One comment on my blog was “You appear to be doing that thing where you write loads, but it is hard to identify any clear points.” Point taken. I’ll see what I can do in this post.

my search for the evidence

I’ve been perplexed by the ideas underpinning the current English education system since my children started encountering problems with it about a decade ago. After a lot of searching, I came to the conclusion that the entire system was lost in a constructivist wilderness. I joined the TES forum to find out more, and discovered that on the whole, teachers weren’t – lost, that is. I came across references to evidence-based educational research and felt hopeful.

Some names were cited; Engelmann, Hirsch, Hattie, Willingham. I pictured a growing body of rigorous research and searched for the authors’ work. Apart from Hattie’s, I couldn’t find much. Willingham was obviously a cognitive psychologist but I couldn’t find his research either. I was puzzled. Most of the evidence seemed to come from magazine articles and a few large-scale studies – notorious for methodological problems. I then heard about Daisy Christodoulou’s book Seven Myths about Education and thought that might give me some pointers. I searched her blog.

In one post, Daisy cites work from the field of information theory by Kirschner, Sweller & Clark, Herb Simon and John Anderson. I was familiar with the last two researchers, but couldn’t open the Simon papers and Anderson’s seemed a bit technical for a general readership. I hadn’t come across the Kirschner, Sweller and Clark reference so I read it. I could see what they were getting at, but thought their reasoning was flawed.

Then it dawned on me. This was the evidence bit of the evidence-based research. It consisted of some early cognitive science/information theory, some large-scale studies and a meta-analysis, together with a large amount of opinion. To me that didn’t constitute a coherent body of evidence. But I was told that there was more to it, which is why I attended the ResearchED conference last weekend. There was more to it, but the substantial body of research didn’t materialise. So where does that leave me?

I still agree with some points that the educational reformers make;

• English-speaking education systems are dominated by constructivist pedagogical approaches
• the implementation of ‘minimal guidance’ approaches has failed to provide children with a good education
• we have a fairly reliable, valid body of knowledge about the world and children should learn about it
• skills tend to be domain-specific
• cognitive science can tell us a lot about how children learn
• the capacity of working memory is limited
• direct instruction is an effective way of teaching.

But I have several reservations that make me uneasy about the education reform ‘movement’.

1. the evidence.

Some is cited frequently. Here’s a summary.

If I’ve understood it correctly, Engelmann and Becker’s DISTAR programme (Direct Instruction System for Teaching Arithmetic and Reading) had far better outcomes for basic maths and reading, higher order cognitive skills (in reading and maths) and responsibility and self-esteem than any other programme in the Project Follow-Through evaluation carried out in 1977.

At around the same time, ED Hirsch had realised that his students’ comprehension of texts was impaired by their poor general knowledge, and in 1983 he published an outline of his concept of what he called ‘cultural literacy’.

A couple of decades later, Daniel Willingham, a cognitive psychologist, started to apply theory from cognitive science to education.

In 2008, John Hattie published Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement – the result of 15 years’ work. The effect sizes Hattie found for various educational factors are ranked here.

Kirschner, Sweller and Clark’s
2006 paper Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching is also often cited. John Sweller developed the concept of ‘cognitive load’ in the 1980s, based on the limited capacity of working memory.

2. the conclusions that can be drawn from the evidence

The DISTAR programme, often referred to as Direct Instruction (capitalised), is clearly very effective for teaching basic maths and literacy. This is an outcome not to be sniffed at, so it would be worth exploring why DISTAR hasn’t been more widely adopted. Proponents of direct instruction often claim it’s because of entrenched ideological opposition; it might also be to do with the fact that it’s a proprietary programme, that teacher input is highly constrained, and that schools have to teach more than basic maths and literacy.

ED Hirsch’s observation that students need prior knowledge before they can comprehend texts involving that knowledge is a helpful one, but has more to say about curriculum design than pedagogy. There are some major issues around all schools using an identical curriculum, who controls the content and how children’s knowledge of the curriculum is assessed.

Daniel Willingham
has written extensively on how findings from cognitive science can be applied to education. Cognitive science is clearly a rich source of useful information. The reason I couldn’t find his research (mainly about procedural memory) appears to be because at some point he changed his middle initial from B to T. I’d assumed it was by someone else.

Although I have doubts about Kirschner Sweller and Clark’s paper, again the contribution from cognitive science is potentially valuable.

John Hattie’s meta-analyses provide some very useful insights into the effectiveness of educational influences.

The most substantial bodies of evidence cited are clearly cognitive science and Hattie’s meta-analyses, which provide a valuable starting point for further exploration of the influences he ranks. Those are my conclusions.

But other conclusions are being drawn – often that the evidence cited above supports the view that direct instruction is the most effective way of teaching and that traditional educational methods (however they are defined) are superior to progressive ones (however they are defined). Those conclusions seem to me to be using the evidence to support beliefs about educational methods, rather than deriving beliefs about educational methods from the evidence.

3. who’s evaluating the evidence?

A key point made by proponents of direct instruction is that students need to have knowledge before they can do anything effective with it. Obviously they do. But this principle appears to be being overlooked by the very people who are emphasizing it.

If you want to understand and apply findings from a meta-analysis you need to be aware of common problems with meta-analyses, how reliable they are, what you need to bear in mind about complex constructs etc. You don’t need to have read everything there is to read about meta-analyses, just to be aware of potential pitfalls. If you want to apply findings from cognitive science, it would help to have at least a broad overview of cognitive science first. That’s because, if you don’t have much prior knowledge, you have no way of knowing how reliable or valid information is. If it’s from a peer-reviewed paper, there’s a good chance it’s reliable because the reviewers would have looked at the theory, the data, the analysis and conclusions. How valid it is (ie how well it maps on to the real world) is another matter. I want to look at some of what ED Hirsch has written to illustrate the point.

Hirsch on psychology and science

Hirsch’s work is often referred to by education reformers. I think he’s right to emphasise the importance of students’ knowledge and I’m impressed by his Core Knowledge framework. There’s now a UK version (slightly less impressive) and his work has influenced the new English National Curriculum. But when I started to check out some of what Hirsch has written I was disconcerted to find that he doesn’t seem to practice what he preaches. In an article in Policy Review he sets out seven ‘reliable general principles’ derived from cognitive science to guide teachers. The principles are sound, even if he has misconstrued ‘chunking’ and views rehearsal as a ‘disagreeable need’.

But Hirsch’s misunderstanding of the history of psychology suggests that not everything he says about psychology might be entirely reliable. He says;

Fifty years ago [the article is dated 2002] psychology was dominated by the guru principle. One declared an allegiance to B.F. Skinner and behaviorism, or to Piaget and stage theory, or to Vygotsky and social theory. Today, by contrast, a new generation of “cognitive scientists,” while duly respectful of these important figures, have leavened their insights with further evidence (not least, thanks to new technology), and have been able to take a less speculative and guru-dominated approach. This is not to suggest that psychology has now reached the maturity and consensus level of solid-state physics. But it is now more reliable than it was, say, in the Thorndike era with its endless debates over “transfer of training.””

This paragraph is riddled with misconceptions. Skinner was indeed an influential psychologist, but behaviourism was controversial – Noam Chomsky was a high profile critic. Piaget was influential in educational circles – but children’s cognitive development formed one small strand of the wide range of areas being investigated by psychologists. Vygotsky’s work has also been influential in education, but it didn’t become widely known in the West until after the publication in 1978 of Mind in Society – a collection of his writings translated into English – so he couldn’t have had ‘guru’ status in psychology in the 1950s. And to suggest that cognitive scientists are ‘duly respectful’ of Skinner, Piaget and Vygotsky as ‘important figures’ in their field, suggests a complete misunderstanding of the roots of cognitive science and of what matters to cognitive scientists. But you wouldn’t be able to question what Hirsch is saying if you had no prior information. And in this article, Hirsch doesn’t support his assertions with references, so you couldn’t check them out.

In a conference address that also forms a chapter in book entitled The Great Curriculum Debate, Hirsch attributes progressive educational methods to the Romantic movement and in turn to religious beliefs, completely overlooking the origins in psychological research of ‘progressive’ educational methodologies and, significantly, the influence of Freud’s work.

In the grand scheme of things, of course, Hirsch’s view of psychology in the 1950s, or his view of the origins of progressive education don’t matter that much. What does matter is that Hirsch himself is seen as something of a guru largely because of his emphasis on students needing to have sound prior knowledge, but here he clearly hasn’t checked out his own.

What’s more important is Hirsch’s view of science. In the last section of his essay Classroom research and cargo cults, entitled ‘on convergence and consensus’, in which he compares classroom research with that from cognitive psychology, he says “independent convergence has always been the hallmark of dependable science“. That’s true in the sense that if several researchers approaching a problem from different directions all come to the same conclusion, they would be reasonably confident that their conclusion was a valid one.

Hirsch illustrates the role of convergence using the example of germ theory. He says “in the nineteenth century, for example, evidence from many directions converged on the germ theory of disease. Once policymakers accepted that consensus, hospital operating rooms, under penalty of being shut down, had to meet high standards of cleanliness.” What’s interesting is that Hirsch slips, almost imperceptibly, from ‘convergence’ into ‘consensus’. In scientific research, convergence is important, but consensus can be extremely misleading because it can be, and often has been, wrong. Ironically, not long before high standards of cleanliness were imposed on hospitals, the consensus had been that cross-contamination theory was wrong, as Semmelweis discovered to his cost. Reliable findings aren’t the same as valid ones.

Hirsch then goes on to say “What policymakers should demand from the [education] research community is consensus.” No they shouldn’t. Consensus can be wrong. What policymakers need to demand from education research is methodological rigour. We already have the relevant expertise, it just needs to be applied to education. Again, if you have no frame of reference against which you can evaluate what Hirsch is saying, you’d be quite likely to assume that he’s right about convergence and consensus – and you’d be none the wiser about the importance of good research design.

what the teachers say

I’m genuinely enthusiastic about teachers wanting to base their practice on evidence. I recognize that this is a work in progress and it’s only just begun. I can quite understand why someone whose teaching has been transformed by a finding from cognitive science might want to share that information as widely as possible. But ironically, some of the teachers involved appear to be doing exactly the opposite of what they recommend teachers do with students.

If you’re not familiar with a knowledge domain, but want to use findings from it, it’s worth getting an overview of it first. This doesn’t involve learning loads of concrete facts, it involves getting someone with good domain knowledge to give you an outline of how it works, so you can see how the concrete facts fit in. It also involves making sure you know what domain-specific skills are required to handle the concrete facts, and whether or not you have them. It also means not making overstated claims. Applying seven principles from cognitive science means you are applying seven principles from cognitive science. That’s all. It’s important to avoid making claims that aren’t supported by the evidence.

What struck me about the supporters of educational reform is that science teachers are noticeable by their absence. Most of the complaints about progressive education seem to relate to English, Mathematics and History. These are all fields that deal with highly abstracted information that is especially vulnerable to constructivist worldviews, so they might have been disproportionately influenced by ‘minimal guidance’ methods. It’s a bit more difficult to take an extreme constructivist approach to physics, chemistry, biology or physical geography because reality tends to intervene quite early on. The irony is that science teachers might be in a better position than teachers of English, Maths or History to evaluate evidence from educational research. And psychology teachers and educational psychologists would have the relevant domain knowledge, which would help avoid reinventing the wheel. I’d recommend getting some of them on board.

2 thoughts on “education reform: evaluating the evidence

  1. I’m trying to identify what concrete points relevant to teaching methods, anybody you refer to has actually got wrong. And I can’t find any. If all you are looking for is arguments from authority, then you won’t find a convincing case from anyone, because arguments from authority are rarely valid arguments. We can always claim about science that it is not the last word, or the whole picture, but that goes without saying. We can also claim that a limited number of individuals dominate particular fields or even particular arguments. But if we have claims supported by cognitive psychology and empirical evidence, and we have claims that are contradicted by cognitive psychology and empirical evidence, which should we go with? Neither set of claims can be proven beyond doubt, but which should be accepted provisionally? I spend a lot of time reading your blogs but when we get rid of the narratives, digressions and claims that somebody is not a sufficient authority, I find little in the way of solid objections to arguments and certainly none that apply if one has already accepted that even the most useful findings of science are provisional.

    • Teaching battleground said;

      “I’m trying to identify what concrete points relevant to teaching methods, anybody you refer to has actually got wrong. And I can’t find any.”

      What education reformers have done is to transfer some information from cognitive science to education. Importing information from knowledge domain A into domain B often improves practice. But because the people working in domain B are not usually experts in domain A, what’s imported is often in the form of simple principles or rules of thumb – they don’t need to know everything about domain A to get something useful from it. In this case, what’s been imported from cognitive science are these principles;

      • working memory has limited capacity
      • chunking information increases the capacity of working memory
      • long-term memory has a much greater capacity
      • rehearsal at intervals improves accessibility of memorised material

      There is nothing wrong with those principles as they stand. As rules of thumb for guiding classroom practice, they are very useful. It’s the ‘as they stand’ bit that’s problematic.

      A knowledge domain is a representation of part of the real world. Cognitive science is about the mechanics of knowledge and thought, education about the transmission of knowledge from one person to another. In the real world, principles and rules of thumb invariably have limitations imposed on them by the other factors they interact with. But when principles and rules of thumb are imported from domain A to domain B, what’s known about the limitations tends to get left behind. The simple principle is what people are after, and what they perceive is making the difference for them.

      Having broken free from the real-world constraints that people working in domain A are aware of, the principles and rules of thumb then take on a life of their own in domain B. As they get passed on from practitioner to practitioner, without reference to domain A, they get added to, subtracted from, reconfigured, until after a few years they bear little resemblance to the original, and are being applied in totally inappropriate ways.

      On top of that, there’s every possibility that in the meantime domain A has moved on, and that the original principle or rule of thumb has since been found to have different limitations, or has completely changed.

      This is why I’m concerned about the simple model of working memory. I don’t think there’s any problem with presenting teachers with a simplified model and telling them it’s simplified, but Willingham’s simplified model is already being presented without the ‘simplified’ proviso. I’ve already heard teachers saying that long-term memory is permanent and doesn’t change. That isn’t the case; over time memories become less accessible and get distorted. They need frequent rehearsal to stay accessible and accurate. Teachers have clearly understood this in relation to what’s learned in school, but it’s misleading to suggest that what’s learned in school, however thoroughly, will remain with people throughout their lives. It won’t. That’s not how long-term memory works.

      Do a few inaccuracies matter? Yes I think they do. The whole point of knowledge is that it represents the real world, or as close as we can get given the limits of our perception. People need that knowledge to be as accurate as possible. It’s not OK to oversimplify or modify knowledge just because it’s convenient.

      Which brings me to:

      “If all you are looking for is arguments from authority, then you won’t find a convincing case from anyone, because arguments from authority are rarely valid arguments. We can always claim about science that it is not the last word, or the whole picture, but that goes without saying. We can also claim that a limited number of individuals dominate particular fields or even particular arguments.”

      The only authority I’m appealing to is that of evidence-based knowledge. I recognise that there are ‘authorities’ in a particular knowledge domain, but I’m well aware that they can be wrong. That’s my point. One of the teachers at the ResearchEd conference in Birmingham gave a presentation that was essentially an exposition of the Hirsch article on classroom research and cargo cults. But there was no critique, no checking what he was saying against other research (quite important since the only full reference in the article is to Feynman’s book), and no mention of the fact that he was demonstrably wrong in his statements about psychology and science.

      Again, do a few inaccuracies matter? Not if they are minor ones. It would obviously be unreasonable to expect everyone to be entirely accurate about everything. But I think it does matter if someone who is being cited as an authority in one knowledge domain is depicting another domain inaccurately, and it matters a great deal if they have fundamentally misunderstood the scientific method, but that misunderstanding isn’t even mentioned by someone propagating the authority’s ideas.

      “But if we have claims supported by cognitive psychology and empirical evidence, and we have claims that are contradicted by cognitive psychology and empirical evidence, which should we go with? Neither set of claims can be proven beyond doubt, but which should be accepted provisionally? I spend a lot of time reading your blogs but when we get rid of the narratives, digressions and claims that somebody is not a sufficient authority, I find little in the way of solid objections to arguments and certainly none that apply if one has already accepted that even the most useful findings of science are provisional.”

      No problem with going with cognitive psychology and empirical evidence. But it should be cognitive psychology and empirical evidence, not out-of-context snippets from them, an oversimplified version of them, or a loose interpretation of them.

      What I’m still struggling with is that the mismatch between the espoused theory of educational ‘reformers’ – that evidence is essential, that students need a sound knowledge of a domain, that concrete facts are important; and the theory-in-use, which doesn’t seem to have clear criteria for what it considers evidence, or to consider relevant domain knowledge important, or to be particularly bothered about the accuracy of all the concrete facts involved.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s