biologically primary and secondary knowledge?

David Geary is an evolutionary psychologist who developed the concept of biologically primary and biologically secondary knowledge, popular with some teachers. I’ve previously critiqued Geary’s ideas as he set them out in a chapter entitled Educating the Evolved Mind. One teacher responded by suggesting I read Geary’s The Origin of Mind because it explained his ideas in more detail. So I did.

Geary’s theory

If I’ve understood correctly, Geary’s argument goes like this:

The human body and brain have evolved over time in response to environmental pressures ranging from climate and diet through to social interaction. For Geary, social interaction is a key driver of evolved brain structures because social interactions can increase the resources available to individuals.

Environmental pressures have resulted in the evolution of brain ‘modules’ specialising in processing certain types of information, such as language or facial features. Information is processed by the modules rapidly, automatically and implicitly, resulting in heuristics (rules of thumb) characteristic of the ‘folk’ psychology, biology and physics that form the default patterns for the way we think. But we are also capable of flexible thought that overrides those default patterns. The flexibility is due to the highly plastic frontal areas of our brain responsible for intelligence. Geary refers to the thinking using the evolved modules as biologically primary, and that involving the plastic frontal areas as biologically secondary.

Chapters 2 & 3 of The Origin of Mind offer a clear, coherent account of Darwinian and hominid evolution respectively. They’d make a great resource for teachers. But when Geary moves on to cognition his model begins to get a little shaky – because it rests on several assumptions.

Theories about evolution of the brain are inevitably speculative because brain tissue decomposes and the fossil record is incomplete. Theories about brain function also involve speculation because our knowledge about how brains work is incomplete. There’s broad agreement on the general principles, but some hypotheses have generated what Geary calls ‘hot debate’. Despite acknowledging the debates, Geary’s model is built on assumptions about which side of the debate is correct. The assumptions involve the modularity of the brain, folk systems, intelligence, and motivation-to-control.

modularity

The general principle of modularity – that there are specific areas of the brain dedicated to processing specific types of information – is not in question. What is less clear is how specialised the modules are. For example, the fusiform face area (FFA) specialises in processing information about faces. But not just faces. It has also been shown to process information about cars, birds, butterflies, chess pieces, Digimon, and novel items called greebles. This raises the question of whether the FFA evolved to process information about faces as such (the Face Specific Hypothesis), or to process information about objects requiring fine-grained discrimination (the Expertise Hypothesis). Geary comes down on the Faces side of the debate on the grounds that the FFA does not “generally respond to other types of objects … that do not have facelike features, except in individuals with inherent sociocognitive deficits, such as autism” (p.141). Geary is entitled to his view, but that’s not the only interpretation of the evidence.

folk systems

The general principle of ‘folk’ systems – evolved forms of thought that result from information being processed rapidly, automatically and implicitly – is also not in question. Geary admits it’s unclear whether the research is “best understood in terms of inherent modular constraints, or as the result of general learning mechanisms” but comes down on the side of children’s thinking being the result of “inherent modular systems”.  I couldn’t find a reference to Eleanor Rosch’s prototype theory developed in the 1970s, which explains folk categories in terms of general learning mechanisms. And it’s regrettable that Rakison & Oakes’ 2008 review of research into how children form categories (that also lends weight to the general learning mechanisms hypothesis) wasn’t published until three years after The Origin of Mind. I don’t know whether either would have prompted Geary to amend his theory.

intelligence

In 1904 Charles Spearman published a review of attempts to measure intellectual ability. He concluded that the correlations between various specific abilities indicated “that there really exists a something that we may provisionally term “General Sensory Discrimination” and similarly a “General Intelligence”” (Spearman p.272).

It’s worth looking at what the specific abilities included. Spearman ranks (p. 276) in order of their correlation with ‘General Intelligence’, performance in: Classics, Common Sense, Pitch Discrimination, French, Cleverness, English, Mathematics, Pitch Discrimination among the uncultured, Music, Light Discrimination and Weight Discrimination.

So, measures of school performance turned out to be good predictors of school performance. The measures of school performance correlated strongly with ‘General Intelligence’ – a construct derived from… the measures of school performance. This tautology wasn’t lost on other psychologists and Spearman’s conclusions received considerable criticism. As Edwin Boring pointed out in 1923, ‘intelligence’ is defined by the content of ‘intelligence’ tests. The correlations between specific abilities and the predictive power of intelligence tests are well-established. What’s contentious is whether they indicate the existence of an underlying ‘general mental ability’.

Geary says the idea that children’s intellectual functioning can be improved is ‘hotly debated’ (p.295). But he appears to look right past the even hotter debate that’s raged since Spearman’s work was published, about whether the construct general intellectual ability (g) actually represents ‘a something’ that ‘really exists’. Geary assumes it does, and also accepts Cattell’s later constructs crystallised and fluid intelligence without question.

Clearly some people are more ‘intelligent’ than others, so the idea of g initially appears valid. But ‘intelligence’ is, ironically, a ‘folk’ construct. It’s a label we apply to a set of loosely defined characteristics – a useful shorthand descriptive term. It doesn’t follow that ‘intelligence’ is a biologically determined ‘something’ that ‘really exists’.

motivation-to-control

The motivation to control relationships, events and resources is a key part of Geary’s theory. He argues that motivation-to-control is an evolved disposition (inherent in the way people think) that manifests itself most clearly in the behaviour of despots – who seek to maximise their control of resources. Curiously, in referring to despots, Geary cites a paper by Herb Simon (Simon, 1990) on altruism (a notoriously knotty problem for evolution researchers). Geary describes an equally successful alternative strategy to despotism, not as altruism but as “adherence to [social] laws and mores”, even though the evidence suggests altruism is an evolved disposition, not merely a behaviour.

Altruism calls into question the control part of the motivation-to-control hypothesis. Many people have a tendency to behave in ways that increase their control of resources, but many tend to collaborate and co-operate instead, strategies that increase individual access to resources, despite reducing individual control over them. The altruism debate is another that’s been going on for decades, but you wouldn’t know that to read Geary.

Then there’s the motivation part. Like ‘intelligence’, ‘motivation’ is a label for a loosely defined bunch of factors that provide incentives for behaviour. ‘Motivation’ is a useful label. But again it doesn’t follow that ‘motivation’ is ‘a something’ that ‘really exists’. The biological mechanisms involved in the motivation to eat or drink are unlikely to be the same as those involved in wanting to marry the boss’s daughter or improve on our personal best for the half-marathon. The first two examples are likely to increase our access to resources; whether they increase our control over them will depend on the circumstances. Geary doesn’t explain the biological mechanism involved.

biologically primary and secondary knowledge

In The Origin of Mind, Geary touches on the idea of biologically primary and secondary competencies and abilities but doesn’t go into detail about their implications for education. Instead, he illustrates the principle by referring to the controlled problem solving used by Charles Darwin and Alfred Wallace in tackling the problem of how different species had arisen.

Geary says that such problem solving requires the inhibition of ‘heuristic-based folk systems’ (p.197), and repeatedly proposes (pp.188, 311, 331, 332) that the prior knowledge of scientific pioneers such as Linnaeus, Darwin and Wallace “arose from evolved folk biological systems…as elaborated by associated academic learning” (p.188). He cites as evidence the assumptions resulting from religious belief made by anatomist and palaeontologist Richard Owen (p.187), and Wallace’s reference to an ‘Overruling Intelligence’ being behind natural selection (p.83). But this proposal is problematic, for three reasons:

The first problem is that some ‘evolved’ folk knowledge is explicit, not implicit. Belief in a deity is undoubtedly folk knowledge; societies all over the world have come up with variations on the concept. But the folk knowledge about religious beliefs is usually culturally transmitted to children, rather than generated by them spontaneously.

Another difficulty is that thinkers such as Linnaeus, Darwin and Wallace had a tendency to be born into scholarly families, so their starting point, even as young children, would not have been merely ‘folk biological systems’. And each of the above had the advantage of previous researchers having already reduced the problem space.

A third challenge is that heuristics aren’t exclusively biologically primary; they can be learned, as Geary points out, via biologically secondary knowledge (p.185).

So if biologically primary knowledge sometimes involves explicit instruction, and biologically secondary knowledge can result in the development of fast, automatic, implicit heuristics, how can we tell which type of knowledge is which?

use of evidence

Geary accepts contentious constructs such as motivation, intelligence and personality (p.319) without question. And he appears to have a rather unique take on concepts such as bounded rationality (p.172), satisficing (p. 173) and schemata (p.186).

In addition, Geary’s evidence is not always contentious; sometimes it’s his conclusions that are tenuous. For example, he predicts that if social competition were a driving force during evolution, “a burning desire to master algebra or Newtonian physics will not be universal or even common. Surveys of the attitudes and preferences of American schoolchildren support this prediction and indicate that they value achievement in sports … much more than achievement in any academic area” (pp.334-5), citing a 1993 paper by Eccles et al. The ‘surveys’ were two studies, the ‘American schoolchildren’ 865 elementary school students, the ‘attitudes and preferences’ competence beliefs and task values, and the ‘academic areas’ math, reading and music. Responses show some statistically significant differences. Geary appears to overegg the evidential pudding somewhat, and to completely look past the possibility that there might be culturally transmitted factors involved.

conclusion

I find Geary’s model perplexing. Most of the key links in it – brain evolution, brain modularity, the heuristics and biases that result in ‘folk’ thinking, motivation and intelligence – involve highly contentious hypotheses.  Geary mentions the ‘hot debates’ but doesn’t go into detail. He simply comes down on one side of the debate and builds his model on the assumption that that side is correct.

He appears to have developed an overarching model of cognition and learning and squeezed the evidence into it, rather than building the model according to the evidence. The problem with the second approach of course, is that if the evidence is inconclusive, you can’t develop an overarching model of cognition and learning without it being highly speculative.

What also perplexes me about Geary’s model is its purpose. Teachers have been aware of the difference between implicit and explicit learning (even if they didn’t call it that) for centuries. It’s useful for them to know about brain evolution and modularity and the heuristics and biases that result in ‘folk’ thinking etc. But teachers can usually spot whether children are learning something apparently effortlessly (implicitly) or whether they need step-by-step (explicit) instruction. That’s essentially why teachers exist. Why do they need yet another speculative educational model?

references

Eccles, J., Wigfield, A., Harold, R.D.,  & Blumenfeld, P. (1993). Age and gender differences in children’s self‐and task perceptions during elementary school, Child Development, 64, 830-847.

Gauthier, I., Tarr, M.J., Anderson, A.W., Skudlarski, P. & Gore, J.C.  (1999). Activation of the middle fusiform ‘face area’ increases with expertise in recognizing novel objects, Nature Neuroscience, 2, 568-573.

Rakison, D.H.  & Oakes L.M. (eds) (2008). Early Category and Concept Development.  Oxford University Press.

Simon, H.A. (1990). A mechanism for social selection and successful altruism. Science, 250, 1665-1668.

Spearman, C.  (1904).  ‘General Intelligence’ objectively determined and measured.  The American Journal of Psychology, 15, 201-292.

 

 

Advertisements

educating the evolved mind: education

The previous two posts have been about David Geary’s concepts of primary and secondary knowledge and abilities; evolved minds and intelligence.  This post is about how Geary applies his model to education in Educating the Evolved Mind.

There’s something of a mismatch between the cognitive and educational components of Geary’s model.  The cognitive component is a range of biologically determined functions that have evolved over several millennia.  The educational component is a culturally determined education system cobbled together in a somewhat piecemeal and haphazard fashion over the past century or so.

The education system Geary refers to is typical of the schooling systems in developed industrialised nations, and according to his model, focuses on providing students with biologically secondary knowledge and abilities. Geary points out that many students prefer to focus on biologically primary knowledge and abilities such as sports and hanging out with their mates (p.52).   He recognises they might not see the point of what they are expected to learn and might need its importance explained to them in terms of social value (p.56). He suggests ‘low achieving’ students especially might need explicit, teacher driven instruction (p.43).

You’d think, if cognitive functions have been biologically determined through thousands of years of evolution, that it would make sense to adapt the education system to the cognitive functions, rather then the other way round. But Geary doesn’t appear to question the structure of the current US education system at all; he accepts it as a given. I suggest that in the light of how human cognition works, it might be worth taking a step back and re-thinking the education system itself in the light of the following principles:

1.communities need access to expertise

Human beings have been ‘successful’, in evolutionary terms, mainly due to our use of language. Language means it isn’t necessary for each of us to learn everything for ourselves from scratch; we can pass on information to each other verbally. Reading and writing allow knowledge to be transmitted across time and space. The more knowledge we have as individuals and communities, the better our chances of survival and a decent quality of life.

But, although it’s desirable for everyone to be proficient reader and writer and to have an excellent grasp of collective human knowledge, that’s not necessary in order for each of us to have a decent quality of life. What each community needs is a critical mass of people with good knowledge and skills.

Also, human knowledge is now so vast that no one can be an expert on everything; what’s important is that everyone has access to the expertise they need, when and where they need it.  For centuries, communities have facilitated access to expertise by educating and training experts (from carpenters and builders to doctors and lawyers) who can then share their expertise with their communities.

2.education and training is not just for school

Prior to the development of mass education systems, most children’s and young people’s education and training would have been integrated into the communities in which they lived. They would understand where their new knowledge and skills fitted into the grand scheme of things and how it would benefit them, their families and others. But schools in mass education systems aren’t integrated into communities. The education system has become its own specialism. Children and young people are withdrawn from their community for many hours to be taught whatever knowledge and skills the education system thinks fit. The idea that good exam results will lead to good jobs is expected to provide sufficient motivation for students to work hard at mastering the school curriculum.  Geary recognises that it doesn’t.

For most of the millennia during which cognitive functions have been developing, children and young people have been actively involved in producing food or making goods, and their education and training was directly related to those tasks. Now it isn’t.  I’m not advocating a return to child labour; what I am advocating is ensuring that what children and young people learn in school is directly and explicitly related to life outside school.

Here’s an example: A highlight of the Chemistry O level course I took many years ago was a visit to the nearby Avon (make-up) factory. Not only did we each get a bag of free samples, but in the course of an afternoon the relevance of all that rote learning of industrial applications, all that dry information about emulsions, fat-soluble dyes, anti-fungal additives etc. suddenly came into sharp focus. In addition, the factory was a major local employer and the Avon distribution network was very familiar to us, so the whole end-to-end process made sense.

What’s commonly referred to as ‘academic’ education – fundamental knowledge about how the world works – is vital for our survival and wellbeing as a species. But knowledge about how the world works is also immensely practical. We need to get children and young people out, into the community, to see how their communities apply knowledge about how the world works, and why it’s important. The increasing emphasis in education in the developed world on paper-and-pencil tests, examination results and college attendance is moving the education system in the opposite direction, away from the practical importance of extensive, robust knowledge to our everyday lives.  And Geary appears to go along with that.

3.(not) evaluating the evidence

Broadly speaking, Geary’s model has obvious uses for teachers.   There’s considerable supporting evidence for a two-phase model of cognition ranging from Fodor’s specialised, stable/general, unstable distinction, to the System 1/System 2 model Daniel Kahnemann describes in Thinking, Fast and Slow. Whether the difference between Geary’s biologically primary and secondary knowledge and abilities is as clear-cut as he claims, is a different matter.

It’s also well established that in order to successfully acquire the knowledge usually taught in schools, children need the specific abilities that are measured by intelligence tests; that’s why the tests were invented in the first place. And there’s considerable supporting evidence for the reliability and predictive validity of intelligence tests. They clearly have useful applications in schools. But it doesn’t follow that what we call intelligence or g (never mind gF or gC) is anything other than a construct created by the intelligence test.

In addition, the fact that there is evidence that supports Geary’s claims doesn’t mean all his claims are true. There might also be considerable contradictory evidence; in the case of Geary’s two-phase model the evidence suggests the divide isn’t as clear-cut as he suggests, and the reification of intelligence has been widely critiqued. Geary mentions the existence of ‘vigorous debate’ but doesn’t go into details and doesn’t evaluate the evidence by actually weighing up the pros and cons.

Geary’s unquestioning acceptance of the concepts of modularity, intelligence and education systems in the developed world, increases the likelihood that teachers will follow suit and simply accept Geary’s model as a given. I’ve seen the concepts of biologically primary and secondary knowledge and abilities, crystallised intelligence (gC) and fluid intelligence (gF), and the idea that students with low gF who struggle with biologically secondary knowledge just need explicit direct instruction, all asserted as if they must be true – presumably because an academic has claimed they are and cited evidence in support.

This absence of evaluation of the evidence is especially disconcerting in anyone who emphasises the importance of teachers becoming research-savvy and developing evidence-based practice, or who posits models like Geary’s in opposition to the status quo. The absence of evaluation is also at odds with the oft cited requirement for students to acquire robust, extensive knowledge about a subject before they can understand, apply, analyse, evaluate or use it creatively. That requirement applies only to school children, it seems.

references

Fodor, J (1983).  The modularity of mind.  MIT Press.

Geary, D (2007).  Educating the evolved mind: Conceptual foundations for an evolutionary educational psychology, in Educating the evolved mind: Conceptual foundations for an evolutionary educational psychology, JS Carlson & JR Levin (Eds). Information Age Publishing.

Kahneman, D (2012).  Thinking, fast and slow.   Penguin.

magic beans, magic bullets and crypto-pathologies

In the previous post, I took issue with a TES article that opened with fidget-spinners and closed with describing dyslexia and ADHD as ‘crypto-pathologies’. Presumably as an analogy with cryptozoology – the study of animals that exist only in folklore. But dyslexia and ADHD are not the equivalent of bigfoot and unicorns.

To understand why, you have to unpack what’s involved in diagnosis.

diagnosis, diagnosis, diagnosis

Accurate diagnosis of health problems has always been a challenge because:

  • Some disorders* are difficult to diagnose. A broken femur, Bell’s palsy or measles are easier to figure out than hypothyroidism, inflammatory bowel disease or Alzheimer’s.
  • It’s often not clear what’s causing the disorder. Fortunately, you don’t have to know the immediate or root causes for successful treatment to be possible. Doctors have made the reasonable assumption that patients presenting with the same signs and symptoms§ are likely to have the same disorder.

Unfortunately, listing the signs and symptoms isn’t foolproof because;

  • some disorders produce different signs and symptoms in different patients
  • different disorders can have very similar signs and symptoms.

some of these disorders are not like the others…

To complicate the picture even further, some signs and symptoms are qualitatively different from the aches, pains, rashes or lumps that indicate disorders obviously located in the body;  they involve thoughts, feelings and behaviours instead. Traditionally, human beings have been assumed to consist of a physical body and non-physical parts such as mind and spirit, which is why disorders of thoughts, feelings and behaviours were originally – and still are – described as mental disorders.

Doctors have always been aware that mind can affect body and vice versa. They’ve also long known that brain damage and disease can affect thoughts, feelings, behaviours and physical health. In the early 19th century, mental disorders were usually identified by key symptoms. The problem was that the symptoms of different disorders often overlapped. A German psychiatrist, Emil Kraepelin, proposed instead classifying mental disorders according to syndromes, or patterns of co-occurring signs and symptoms. Kraepelin hoped this approach would pave the way for finding the biological causes of disorders. (In 1906, Alois Alzheimer found the plaques that caused the dementia named after him, while he was working in Kraepelin’s lab.)

Kraepelin’s approach laid the foundations for two widely used modern classification systems for mental disorders; the Diagnostic and Statistical Manual of Mental Disorders, published by the American Psychiatric Association, currently in its 5th edition (DSM V), and the International Classification of Diseases Classification of Mental and Behavioural Disorders published by the World Health Organisation, currently in its 10th edition (ICD-10).

Kraepelin’s hopes for his classification system have yet to be realised. That’s mainly because the brain is a difficult organ to study. You can’t poke around in it without putting your patient at risk. It’s only in the last few decades that scanning techniques have enabled researchers to look more closely at the structure and function of the brain, and the scans require interpretation –  brain imaging is still in its infancy.

you say medical, I say experiential

Kraepelin’s assumptions about distinctive patterns of signs and symptoms, and about their biological origins, were reasonable ones. His ideas, however, were almost the polar opposite to those of his famous contemporary, Sigmund Freud, who located the root causes of mental disorders in childhood experience. The debate has raged ever since. The dispute is due to the plasticity of the brain.  Brains change in structure and function over time and several factors contribute to the changes;

  • genes – determine underlying structure and function
  • physical environment e.g. biochemistry, nutrients, toxins – affects structure and function
  • experience – the brain processes information, and information changes the brain’s physical structure and biochemical function.

On one side of the debate is the medical model; in essence, it assumes that the causes of mental disorders are primarily biological, often due to a ‘chemical imbalance’. There’s evidence to support this view; medication can improve a patient’s symptoms. The problem with the medical model is that it tends to assume;

  • a ‘norm’ for human thought, feelings and behaviours – disorders are seen as departures from that norm
  • the cause of mental disorders is biochemical and the chemical ‘imbalance’ is identified (or not) through trial-and-error – errors can be catastrophic for the patient.
  • the cause is located in the individual.

On the other side of the debate is what I’ll call the experiential model (often referred to as anti-psychiatry or critical psychiatry). In essence it assumes the causes of unwanted thoughts, feelings or behaviours are primarily experiential, often due to adverse experiences in childhood. The problem with that model is that it tends to assume;

  • the root causes are experiential and not biochemical
  • the causes are due to the individual’s response to adverse experiences
  • first-hand reports of early adverse experiences are always reliable, which they’re not.

labels

Kraepelin’s classification system wasn’t definitive – it couldn’t be, because no one knew what was causing the disorders. But it offered the best chance of identifying distinct mental health problems – and thence their causes and treatments. The disorders identified in Kraepelin’s system, the DSM and ICD, were – and most still are – merely labels given to clusters of co-occurring signs and symptoms.  People showing a particular cluster are likely to share the same underlying biological causes, but that doesn’t mean they do share the same underlying causes or that the origin of the disorder is biological.

This is especially true for signs and symptoms that could have many causes. There could be any number of reasons for someone hallucinating, withdrawing, feeling depressed or anxious – or having difficulty learning to read or maintain attention.  They might not have a medical ‘disorder’ as such. But you wouldn’t know that to read through the disorders listed in the DSM or ICD. They all look like bona fide, well-established medical conditions, not like labels for bunches of symptoms that sometimes co-occur and sometimes don’t, and that have a tendency to appear or disappear with each new edition of the classification system.  That brings us to the so-called ‘crypto-pathologies’ referred to in the TES article.

Originally, terms like dyslexia were convenient and legitimate shorthand labels for specific clusters of signs or symptoms. Dyslexia means difficulty with reading, as distinct from alexia which means not being able to read at all; both problems can result from stroke or brain damage. Similarly, autism was originally a shorthand term for the withdrawn state that was one of the signs of schizophrenia – itself a label.  Delusional parasitosis is also a descriptive label (the parasites being what’s delusional, not the itching).

reification

What’s happened over time is that many of these labels have become reified – they’ve transformed from mere labels into disorders widely perceived as having an existence independent of the label. Note that I’m not saying the signs and symptoms don’t exist. There are definitely children who struggle with reading regardless of how they’ve been taught; with social interaction regardless of how they’ve been brought up; and with maintaining focus regardless of their environment. What I am saying is that there might be different causes, or multiple causes, for clusters of very similar signs and symptoms.  Similar signs and symptoms don’t mean that everybody manifesting those signs and symptoms has the same underlying medical disorder –  or even that they have a medical disorder at all.

The reification of labels has caused havoc for decades with research. If you’ve got a bunch of children with different causes for their problems with reading, but you don’t know what the different causes are so you lump all the children together according to their DSM label; or another bunch with different causes for their problems with social interaction but lump them all together; or a third bunch with different causes for their problems maintaining focus, but you lump them all together; you are not likely to find common causes in each group for the signs and symptoms.  It’s this failure to find distinctive features at the group level that has been largely responsible for claims that dyslexia, autism or ADHD ‘don’t exist’, or that treatments that have evidently worked for some individuals must be spurious because they don’t work for other individuals or for the heterogeneous group as a whole.

crypto-pathologies

Oddly, in his TES article, Tom refers to autism as an ‘identifiable condition’ but to dyslexia and ADHD as ‘crypto-pathologies’ even though the diagnostic status of autism in the DSM and ICD is on a par with that of ADHD, and with ‘specific learning disorder with impairment in reading‘ with dyslexia recognised as an alternative term (DSM), or ‘dyslexia and alexia‘ (ICD).  Delusional parasitosis, despite having the same diagnostic status and a plausible biological mechanism for its existence, is dismissed as ‘a condition that never was’.

Tom is entitled to take a view on diagnosis, obviously. He’s right to point out that reading difficulties can be due to lack of robust instruction, and inattention can be due to the absence of clear routines. He’s right to dismiss faddish simplistic (but often costly) remedies. But the research is clear that children can have difficulties with reading due to auditory and/or visual processing impairments (search Google scholar for ‘dyslexia visual auditory’), that they can have difficulties maintaining attention due to low dopamine levels – exactly what Ritalin addresses (Iversen, 2006), or that they can experience intolerable itching that feels as if it’s caused by parasites.

But Tom doesn’t refer to the research, and despite provisos such as acknowledging that some children suffer from ‘real and grave difficulties’ he effectively dismisses some of those difficulties as crypto-pathologies and implies they can be fixed by robust teaching and clear routines  –  or that they are just imaginary.  There’s a real risk, if the research is by-passed, of ‘robust teaching’ and ‘clear routines’ becoming the magic bullets and magic beans he rightly despises.

Notes

*Disorder implies a departure from the norm.  At one time, it was assumed the norm for each species was an optimal set of characteristics.  Now, the norm is statistically derived, based on 95% of the population.

§ Technically, symptoms are indicators of a disorder experienced only by the patient and signs are detectable by others.  ‘Symptoms’ is often used to include both.

Reference

Iversen, L (2006).  Speed, Ecstasy, Ritalin: The science of amphetamines.  Oxford University Press.

white knights and imaginary dragons: Tom Bennett on fidget-spinners

I’ve crossed swords – or more accurately, keyboards – with Tom Bennett, the government’s behaviour guru tsar adviser, a few times, mainly about learning styles. And about Ken Robinson. Ironic really, because broadly speaking we’re in agreement. Ken Robinson’s ideas about education are woolly and often appear to be based on opinion rather than evidence, and there’s clear evidence that teachers who use learning styles, thinking hats and brain gym probably are wasting their time. Synthetic phonics helps children read and whole school behaviour policies are essential for an effective school and so on…

My beef with Tom has been his tendency to push his conclusions further than the evidence warrants. Ken Robinson is ‘the butcher given a ticker tape parade by the National Union of Pigs‘.  Learning Styles are ‘the ouija board of serious educational research‘.  What raised red flags for me this time is a recent TES article by Tom prompted by the latest school-toy fad ‘fidget-spinners’.

fidget-spinners

Tom begins with claims that fidget-spinners can help children concentrate. He says “I await the peer-reviewed papers from the University of Mickey Mouse confirming these claims“, assuming that he knows what the evidence will be before he’s even seen it.  He then introduces the idea that ‘such things’ as fidget-spinners might help children with an ‘identifiable condition such as autism or sensory difficulties’, and goes on to cite comments from several experts about fidget-spinners in particular and sensory toys in general. We’re told “…if children habitually fidget, the correct path is for the teacher to help the child to learn better behaviour habits, unless you’ve worked with the SENCO and the family to agree on their use. The alternative is to enable and deepen the unhelpful behaviour. Our job is to support children in becoming independent, not cripple them with their own ticks [sic]”.

If a child’s fidgeting is problematic, I completely agree that a teacher’s first course of action should be to help them stop fidgeting, although Tom offers no advice about how to do this. I’d also agree that the first course of action in helping a fidgety child shouldn’t be to give them a fidget-toy.

There’s no question that children who just can’t seem to sit still, keep their hands still, or who incessantly chew their sleeves, are seeking sensory stimulation, because that’s what those activities are – by definition. It doesn’t follow that allowing children to walk about, or use fidget or chew toys will ‘cripple them with their own ticks’. These behaviours are not tics, and usually extinguish spontaneously over time. If they’re causing disruption in the classroom, questions need to be asked about school expectations and the suitability of the school provision for the child, not about learning unspecified ‘better behaviour habits’.

mouthwash

Tom then devotes an entire paragraph to, bizarrely, Listerine. His thesis is that sales of antiseptic mouthwash soared due to an advertising campaign persuading Americans that halitosis was a serious social problem. His evidence is a blogpost by Sarah Zhang, a science journalist.  Sarah’s focus is advertising that essentially invented problems to be cured by mouthwash or soap. Neither she nor Tom mention the pre-existing obsession with cleanliness that arose from the discovery – prior to the discovery of antibiotics – that a primary cause of death and debility was bacterial infections that could be significantly reduced by the use of alcohol rubs, boiling and soap.

itchy and scratchy

The Listerine advertising campaign leads Tom to consider ‘fake or misunderstood illnesses’ that he describes as ‘charlatan’. His examples are delusional parasitosis (people believe their skin is itching because it’s infested with parasites) and Morgellon’s (belief that the itching is caused by fibres). Tom says “But there are no fibres or parasites. It’s an entirely psycho-somatic condition. Pseudo sufferers turn up at their doctors scratching like mad, some even cutting themselves to dig out the imaginary threads and crypto-bugs. Some doctors even wearily prescribe placebos and creams that will relieve the “symptoms”. A condition that never was, dealt with by a cure that won’t work. Spread as much by belief as anything else, like fairies.”

Here, Tom is pushing the evidence way beyond its limits. The fact that the bugs or fibres are imaginary doesn’t mean the itching is imaginary. The skin contains several different types of tactile receptor that send information to various parts of the brain. The tactile sensory system is complex so there are several points at which a ‘malfunction’ could occur.  The fact that busy GPs – who for obvious reasons don’t have the time or resources to examine the functioning of a patient’s neural pathways at molecular level – wearily prescribe a placebo, says as much about the transmission of medical knowledge in the healthcare system as it does about patients’ beliefs.

crypto-pathologies

Tom refers to delusional parasitosis and Morgellon’s as ‘crypto-pathologies’ – whatever that means – and then introduces us to some crypto-pathologies he claims are encountered in school; dyslexia and ADHD. As he points out dyslexia and ADHD are indeed labels for ‘a collection of observed symptoms’. He’s right that some children with difficulty reading might simply need good reading tuition, and those with attention problems might simply need a good relationship with their teacher and clear routines. As he points out “…our diagnostic protocol is often blunt. Because we’re unsure what it is we’re diagnosing, and it becomes an ontological problem“.  He then says “This matters when we pump children with drugs like Ritalin to stun them still.

Again, some of Tom’s claims are correct but others are not warranted by the evidence. In the UK, Ritalin is usually prescribed by a paediatrician or psychiatrist after an extensive assessment of the child, and its effects should be carefully monitored. It’s a stimulant that increases available levels of dopamine and norepinephrine and it often enhances the ability to concentrate. It isn’t ‘pumped into’ children and it doesn’t ‘stun them still’, In the UK at least, NICE guidelines indicate it should be used as a last resort. The fact that its use has doubled in the last decade is a worrying trend. This is more likely to be due to the crisis in child and adolescent mental health services, than to an assumption that all attention problems in children are caused by a supposed medical condition we call ADHD.

Tom, rightly, targets bullshit. He says it matters because “many children suffer from very real and very grave difficulties, and it behoves us as their academic and social guardians to offer support and remedy when we can”. Understandably he wants to drive his point home. But superficial analysis and use of hyperbole risk real and grave difficulties being marginalised at best and ridiculed at worst by teachers who don’t have the time/energy/inclination to check out the detail of what he claims.

Specialist education, health and care services for children have been in dire straits for many years and the situation isn’t getting any better. This means teachers are likely to have little information about the underlying causes of children’s difficulties in school. If teachers take what Tom says at face value, there’s a real risk that children with real difficulties, whether they need to move their fingers or chew in order to concentrate, experience unbearable itching, struggle to read because of auditory, visual or working memory impairments, or have levels of dopamine that prevent them from concentrating, will be seen by some as having ‘crypto-conditions’ that can be resolved by good teaching and clear routines. If they’re not resolved, then the condition must be ‘psycho-somatic’.  Using evidence to make some points, but ignoring it to make others means the slings and arrows Tom hurls at the snake-oil salesmen and white knights galloping to save us from imaginary dragons are quite likely to be used as ammunition against the very children he seeks to help.

a modern day trivium

In the two previous posts, I’ve criticised Martin Robinson’s argument that traditional and progressive education are mutually exclusive approaches characterised by single core values; subject centred and child centred, respectively.

Martin describes himself as an “educationalist with an interest in culture, politics, creativity, and the Liberal Arts (especially grammar, dialectic and rhetoric)”. Grammar, logic and rhetoric are the three strands of the mediaeval trivium and Martin’s educational consultancy and his blog are called Trivium 21C. In response to my comments, he suggested I produce a graphical representation of my understanding of the trivium.

liberal arts, trivium and quadrivium

In Ancient Greece and Rome, the liberal arts were the knowledge and skills it was considered essential for a free man to learn in order to participate in civic society. The liberal arts were revived during the reign of Charlemagne in the 8th century, in an effort to improve educational and cultural standards across Western Europe. Seven subjects were studied; grammar, logic and rhetoric made up the foundational trivium, and the quadrivium consisted of arithmetic, geometry, astronomy and music.

The trivium essentially trained students to think, and the quadrivium gave them the opportunity to apply their thinking to mathematical concepts (considered fundamental to all knowledge by the Greek philosophers). The seven liberal arts formed the foundation that enabled students to proceed to study theology, medicine or law.

Up until the early 19th century, the body of collective human knowledge was relatively small.   It was possible for a well-educated person to master all of it.   In order to acquire the knowledge, and to understand and apply it, you’d have to learn Latin and probably Greek, and also how scholars (who would have written in Latin) reasoned. The trivium made explicit the structure of language, how language was used to reason, and how to explain and persuade using language.

Since the early 19th century our collective knowledge has expanded enormously and much of that knowledge is recorded in English. There are good reasons why English-speaking students should learn the structure of their native language, how to reason in it, and how to use it to explain and persuade. But those skills wouldn’t be much use without the knowledge to apply them to.

I can see how those principles could be applied to our current body of knowledge, and that’s what I’ve mapped out below.Slide1

Grammar would make explicit the structure of the knowledge (including the structure of language). Logic would make reasoning explicit – and common errors and biases in thinking. (Martin replaces logic with dialectic, a process by which different parties seek to arrive at the truth through reasoned discussion with each other.) Rhetoric would make explicit the representation of knowledge, including how people conceptualise it. Incidentally, the body of knowledge has a fuzzy boundary because although much of it is reliable, some is still uncertain.

modern liberal arts and cultural literacy

Many modern colleges and universities offer liberal arts courses, although what’s entailed varies widely. Whatever the content, the focus of liberal arts is on preparing the student for participation in civic society, as distinct from professional, vocational or technical training.

So… I can see the point of the trivium in its original context. And how the principles of the trivium could be applied to the body of knowledge available in the 21st century. Those principles would provide a practical way of ensuring students had a thorough understanding of the knowledge they were applied to.

But… I do have some concerns about using the trivium to do that. The emphasis of the trivium and of liberal arts education, is on language. Language is the primary vehicle for ideas, so there are very good reasons for students mastering language and its uses. And the purpose of a liberal arts education is to prepare students for life, rather than just for work. There are good reasons for that too; human beings are obviously much more than economic units.

However, language and the ideas it conveys also appears to be the end-point of education for liberal arts advocates, rather than just a means to an end. The content of the education is frequently described as ‘the best which has been thought or said’ (Arnold, 1869), and the purpose to enable students to participate in the ‘conversation of mankind’ (Oakeshott, 1962).

The privileging of words and abstract ideas over the nitty-gritty of everyday life is a characteristic of liberal arts education that runs from Plato through the mediaeval period to the modern day. Plato was primarily concerned with the philosopher king and the philosophers who debated with him, not with people who grew vegetables, made copper pots or traded olive oil.   Charlemagne’s focus was on making sure priests could read the Vulgate and that there were enough skilled scribes to keep records, not in improving technology, or the fortunes of the wool industry.

This dualistic rift still permeates thinking about education as evidenced by the ongoing debate about academic v vocational education. Modern-day liberal arts advocates favour the academic approach because, rightly, they see education as more than preparation for work.   Their emphasis, instead, is on cultural literacy. Cultural literacy is important for everybody because it gives access to ideas. However, the flow of information needs to be in two directions, not just one.

Recent events suggest that policy-makers who attended even ‘the best’ private schools, where cultural literacy was highly valued, have struggled to generate workable solutions to the main challenges facing the human race; the four identified by Capra and Luisi (2014) are globalisation, climate change, agriculture, and sustainable design. The root causes and the main consequences of such challenges involve the lowest, very concrete levels that would be familiar to ancient Greek farmers, coppersmiths and merchants, to mediaeval carpenters and weavers, and to those who work in modern factories, but might be unfamiliar to philosophers, scholars or politicians who could rely on slaves or servants.

An education that equips people for life rather than work does not have to put language and ideas on a pedestal; we are embodied beings that live in a world that is uncompromisingly concrete and sometimes sordidly practical. An all-round education will involve practical science, technology and hands-on craft skills, not to prepare students for a job, but so they know how the world works.  It will not just prepare them for participating in conversations.

references

Arnold, M (1869).  Culture and Anarchy.  Accessed via Project Gutenberg http://www.gutenberg.org/cache/epub/4212/pg4212-images.html

Capra, F and Luisi, PL (2014).  The Systems View of Life, Cambridge University Press (p. 394)

Oakeshott, M (1962).”The Voice of Poetry in the Conversation of Mankind” in Rationalism in Politics and Other Essays. London: Methuen, 197-247. Accessed here http://english.ttu.edu/kairos/2.1/features/brent/oakeshot.htm

the Tiger Teachers and cognitive science

Cognitive science is a key plank in the Tiger Teachers’ model of knowledge. If I’ve understood it properly the model looks something like this:

Cognitive science has discovered that working memory has limited capacity and duration, so pupils can’t process large amounts of novel information. If this information is secured in long-term memory via spaced, interleaved practice, students can recall it instantly whenever they need it, freeing up working memory for thinking.

What’s wrong with that? Nothing, as it stands. It’s what’s missing that’s the problem.

Subject knowledge

One of the Tiger Teachers’ beefs about the current education system is its emphasis on transferable skills. They point out that skills are not universally transferable, many are subject-specific, and in order to develop expertise in higher-level skills novices need a substantial amount of subject knowledge. Tiger Teachers’ pupils are expected to pay attention to experts (their teachers) and memorise a lot of facts before they can comprehend, apply, analyse, synthesise or evaluate. The model is broadly supported by cognitive science and the Tiger Teachers apply it rigorously to children. But not to themselves, it seems.

For most Tiger Teachers cognitive science will be an unfamiliar subject area. That makes them (like most of us) cognitive science novices. Obviously they don’t need to become experts in cognitive science to apply it to their educational practice, but they do need the key facts and concepts and a basic overview of the field. The overview is important because they need to know how the facts fit together and the limitations of how they can be applied.   But with a few honourable exceptions (Daisy Christodoulou, David Didau and Greg Ashman spring to mind – apologies if I’ve missed anyone out), many Tiger Teachers don’t appear to have even thought about acquiring expertise, key facts and concepts or an overview. As a consequence facts are misunderstood or overlooked, principles from other knowledge domains are applied inappropriately, and erroneous assumptions made about how science works. Here are some examples (page numbers refer to Battle Hymn of the Tiger Teachers):

It’s a fact…

Teachers’ brains work exactly the same way as pupils’” (p.177). No they don’t. Cognitive science (ironically) thinks that children’s brains begin by forming trillions of connections (synapses). Then through to early adulthood, synapses that aren’t used get pruned, which makes information processing more efficient. (There’s a good summary here.)  Pupils’ brains are as different to teachers’ brains as children’s bodies are different to adults’ bodies. Similarities don’t mean they’re identical.

Then there’s working memory.As the cognitive scientist Daniel Willingham explains, we learn by transferring knowledge from the short-term memory to the long term memory” (p177). Well, kind of – if you assume that what Willingham explicitly describes as “just about the simplest model of the mind possible”  is an exhaustive model of memory. If you think that, you might conclude, wrongly, “the more knowledge we have in long-term memory, the more space we have in our working memory to process new information” (p.177). Or that “information cannot accumulate into long-term memory while working memory is being used” (p.36).

Long-term memory takes centre stage in the Tiger Teachers’ model of cognition. The only downside attributed to it is our tendency to forget things if we don’t revisit them (p.22). Other well-established characteristics of long-term memory – its unreliability, errors and biases – are simply overlooked, despite Daisy Christodoulou’s frequent citation of Daniel Kahneman whose work focused on those flaws.

With regard to transferable skills we’re told “cognitive scientist Herb Simon and his colleagues have cast doubt on the idea that there are any general or transferable cognitive skills” (p.17), when what they actually cast doubt on is the ideas that all skills are transferable or that none are.

The Michaela cognitive model is distinctly reductionist; “all there is to intelligence is the simple accrual and tuning of many small units of knowledge that in total produce complex cognition” (p.19). Then there’s “skills are simply just a composite of sequential knowledge – all skills can be broken down to irreducible pieces of knowledge” (p.161).

The statement about intelligence is a direct quote from John Anderson’s paper ‘A Simple Theory of Complex Cognition’ but Anderson isn’t credited, so you might not know he was talking about simple encodings of objects and transformations, and that by ‘intelligence’ he means how ants behave rather than IQ. I’ve looked at Daisy Christodoulou’s interpretation of Anderson’s model here.

The idea that intelligence and skills consist ‘simply just’ of units of knowledge ignores Anderson’s procedural rules and marginalises the role of the schema – the way people configure their knowledge. Joe Kirby mentions “procedural and substantive schemata” (p. 17), but seems to see them only in terms of how units of knowledge are configured for teaching purposes; “subject content knowledge is best organised into the most memorable schemata … chronological, cumulative schemata help pupils remember subject knowledge in the long term” (p.21). The concept of schemata as the way individuals, groups or entire academic disciplines configure their knowledge, that the same knowledge can be configured in different ways resulting in different meanings, or that configurations sometimes turn out to be profoundly wrong, doesn’t appear to feature in the Tiger Teachers’ model.

Skills: to transfer or not to transfer?

Tiger Teachers see higher-level skills as subject-specific. That hasn’t stopped them applying higher-level skills from one domain inappropriately to another. In her critique of Bloom’s taxonomy, Daisy Christodoulou describes it as a ‘metaphor’ for the relationship between knowledge and skills. She refers to two other metaphors; ED Hirsch’s scrambled egg and Joe Kirby’s double helix (Seven Myths p.21).  Daisy, Joe and ED teach English, and metaphors are an important feature in English literature. Scientists do use metaphors, but they use analogies more often, because in the natural world patterns often repeat themselves at different levels of abstraction. Daisy, Joe and ED are right to complain about Bloom’s taxonomy being used to justify divorcing skills from knowledge. And the taxonomy itself might be wrong or misleading.   But it is a taxonomy and it is based on an important scientific concept – levels of abstraction – so should be critiqued as such, not as if it were a device used by a novelist.

Not all evidence is equal

A major challenge for novices is what criteria they can use to decide whether or not factual information is valid. They can’t use their overview of a subject area if they don’t have one. They can’t weigh up one set of facts against another if they don’t know enough facts. So Tiger Teachers who are cognitive science novices have to fall back on the criteria ED Hirsch uses to evaluate psychology – the reputation of researchers and consensus. Those might be key criteria in evaluating English literature, but they’re secondary issues for scientific research, and for good reason.

Novices then have to figure out how to evaluate the reputation of researchers and consensus. The Tiger Teachers struggle with reputation. Daniel Willingham and Paul Kirschner are cited more frequently than Herb Simon, but with all due respect to Willingham and Kirschner, they’re not quite in the same league. Other key figures don’t get a mention.  When asked what was missing from the Tiger Teachers’ presentations at ResearchEd, I suggested, for starters, Baddeley and Hitch’s model of working memory. It’s been a dominant model for 40 years and has the rare distinction of being supported by later biological research. But it’s mentioned only in an endnote in Willingham’s Why Don’t Students Like School and in Daisy’s Seven Myths about Education. I recommended inviting Alan Baddeley to speak at ResearchEd – he’s a leading authority on memory after all.   One of the teachers said he’d never even heard of him. So why was that teacher doing a presentation on memory at a national education conference?

The Tiger Teachers also struggle with consensus. Joe Kirby emphasises the length of time an idea has been around and the number of studies that support it (pp.22-3), overlooking the fact that some ideas can dominate a field for decades, be supported by hundreds of studies and then turn out to be profoundly wrong; theories about how brains work are a case in point.   Scientific theory doesn’t rely on the quantity of supporting evidence; it relies on an evaluation of all relevant evidence – supporting and contradictory – and takes into account the quality of that evidence as well.  That’s why you need a substantial body of knowledge before you can evaluate it.

The big picture

For me, Battle Hymn painted a clearer picture of the Michaela Community School than I’d been able to put together from blog posts and visitors’ descriptions. It persuaded me that Michaela’s approach to behaviour management is about being explicit and consistent, rather than simply being ‘strict’. I think having a week’s induction for new students and staff (‘bootcamp’) is a great idea. A systematic, rigorous approach to knowledge is vital and learning by rote can be jolly useful. But for me, those positives were all undermined by the Tiger Teachers’ approach to their own knowledge.  Omitting key issues in discussions of Rousseau’s ideas, professional qualifications or the special circumstances of schools in coastal and rural areas, is one thing. Pontificating about cognitive science and then ignoring what it says is quite another.

I can understand why Tiger Teachers want to share concepts like the limited capacity of working memory and skills not being divorced from knowledge.  Those concepts make sense of problems and have transformed their teaching.  But for many Tiger Teachers, their knowledge of cognitive science appears to be based on a handful of poorly understood factoids acquired second or third hand from other teachers who don’t have a good grasp of the field either. Most teachers aren’t going to know much about cognitive science; but that’s why most teachers don’t do presentations about it at national conferences or go into print to share their flimsy knowledge about it.  Failing to acquire a substantial body of knowledge about cognitive science makes its comprehension, application, analysis, synthesis and evaluation impossible.  The Tiger Teachers’ disregard for principles they claim are crucial is inconsistent, disingenuous, likely to lead to significant problems, and sets a really bad example for pupils. The Tiger Teachers need to re-write some of the lyrics of their Battle Hymn.

References

Birbalsingh, K (2016).  Battle Hymn of the Tiger Teachers: The Michaela Way.  John Catt Educational.

Christodoulou, D (2014).  Seven Myths about Education.  Routledge.

getting the PISA scores under control

The results of the OECD’s 2015 Programme for International Student Assessment (PISA) were published a couple of weeks ago. The PISA assessment has measured the performance of 15 year-olds in Reading, Maths and Science every three years since 2000. I got the impression that teachers and academics (at least those using social media) were interested mainly in various aspects of the analysis. The news media, in contrast, focussed on the rankings. So did the OECD and politicians according to the BBC website. Andreas Schleicher of the OECD mentions Singapore ‘getting further ahead’ and John King US Education Secretary referred to the US ‘losing ground’.

What they are talking about are some single-digit changes in scores of almost 500 points. Although the PISA analysis might be informative, the rankings tell us very little. No one will get promoted or relegated as a consequence of their position in the PISA league table. Education is not football. What educational performance measures do have in common with all other performance measures – from football to manufacturing – is that performance is an outcome of causal factors. Change the causal factors and the performance will change.

common causes vs special causes

Many factors impact on performance. Some fluctuations are inevitable because of the variation inherent in raw materials, climatic conditions, equipment, human beings etc. Other changes in performance occur because a key causal factor has changed significantly. The challenge is in figuring out whether fluctuations are due to variation inherent in the process, or whether they are due to a change in the process itself – referred to as common causes and special causes, respectively.

The difference between common causes and special causes is important because there’s no point spending time and effort investigating common causes. Your steel output might have suffered because of a batch of inferior iron ore, your team might have been relegated because two key players sustained injuries, or your PISA score might have fallen a couple of points  due to a flu epidemic just before the PISA tests. It’s impossible to prevent such eventualities and even if you could, some other variation would crop up instead. However, if performance has improved or deteriorated following a change in supplier, strategy or structure you’d want to know whether or not that special cause has had a real impact.

spotting the difference

This was the challenge facing Walter A Shewhart, a physicist, engineer and statistician working for the Western Electric Company in the 1920s. Shewhart figured out a way of representing variations in performance so that quality controllers could see at a glance whether the variation was due to common causes or special causes. The representation is generally known as a control chart. I thought it might be interesting to plot some PISA results as a control chart, to see if changes in scores represented a real change or whether they were the fluctuations you’d expect to see due to variation inherent in the process.

If I’ve understood Shewhart’s reasoning correctly, it goes like this: Even if you don’t change your process, fluctuations in performance will occur due to the many different factors that impact on the effectiveness of your process. In the case of the UK’s PISA scores, each year similar students have learned and been assessed on very similar material, so the process remains unchanged; what the PISA scores measure is student performance.   But student performance can be affected by a huge number of factors; health, family circumstances, teacher recruitment, changes to the curriculum a decade earlier etc.

For statistical purposes, the variation caused by those multiple factors can be treated as random. (It isn’t truly random, but for most intents and purposes can be treated as if it is.) This means that over time, UK scores will form a normal distribution – most will be close to the mean, a few will be higher and a few will be lower. And we know quite a bit about the features of normal distributions.

Shewhart came up with a formula for calculating the upper and lower limits of the variation you’d expect to see as a result of common causes. If a score falls outside those limits, it’s worth investigating because it probably indicates a special cause. If it doesn’t, it isn’t worth investigating, because it’s likely to be due to common causes rather than a change to the process. Shewhart’s method is also useful for finding out whether or not an intervention has made a real difference to performance.  Donald Wheeler, in Understanding Variation: The key to managing chaos, cites the story of a manager spotting a change in performance outside the control limits and discovering it was due to trucks being loaded differently without the supervisor’s knowledge.

getting the PISA scores under control

I found it surprisingly difficult, given the high profile of the PISA results, to track down historical data and I couldn’t access it via the PISA website – if anyone knows of an accessible source I’d be grateful. Same goes for any errors in my calculations.  I decided to use the UK’s overall scores for Mathematics as an example. In 2000 and 2003 the UK assessments didn’t meet the PISA criteria, so the 2000 score is open to question and the 2003 score was omitted from the tables.

I’ve followed the method set out in Donald Wheeler’s book, which is short, accessible and full of examples. At first glance the formulae might look a bit complicated, but the maths involved is very straightforward. Year 6s might enjoy applying it to previous years’ SATs results.

Step 1: Plot the scores and find the mean.

year 2000* 2003* 2006 2009 2012 2015 mean (Xbar§)
UK maths score 529 495 492 494 492 500.4

Table 1: UK maths scores 2000-2015

* In 2000 and 2003 the UK assessments didn’t meet the PISA criteria, so the 2000 score is open to question and the 2003 score was omitted from the results.

§  I was chuffed when I figured out how to type a bar over a letter (the symbol for mean) but it got lost in translation to the blog post.

pisa-fig-1Fig 1: UK Maths scores and mean score

Step 2: Find the moving range (mR) values and calculate the mean.

The moving range is the differences between consecutive scores, referred to as mR values.

year 2000 2003 2006 2009 2012 2015 mean

(R bar)

UK maths score 529 495 492 494 492
mR values 34 3 2 2 10.25

Table 2: moving range (mR values) 2000-2015

pisa-fig-2Fig 2: Differences between consecutive scores (mR values)

Step 3: Calculate the Upper Control Limit for the mR values (UCLR).

To do this we multiply the mean of the mR values (Rbar) by 3.27.

UCLR = 3.27 x Rbar = 3.27 x 10.25 = 33.52

pisa-fig-3Fig 3: Differences between scores (mR values) showing upper control limit (UCLR)

Step 4: Calculate the Upper Natural Process Limit (UNPL) for the individual scores using the formula UNPL = Xbar + (2.66 x Rbar )

UNPL = Xbar + (2.66 x Rbar ) = 500.4 + (2.66 x 10.25) = 500.4 + 27.27 = 527.67

Step 5: Calculate the Lower Natural Process Limit (LNPL) for the individual scores using the formula LNPL = Xbar – (2.66 x Rbar )

LNPL = Xbar – (2.66 x Rbar) = 500.4 – (2.66 x 10.25) = 500.4 – 27.27 = 473.13

We can now plot the UK’s Maths scores showing the upper and lower natural process limits – the limits of the variation you’d expect to see as a result of common causes.

pisa-fig-4Fig 4: UK Maths scores showing upper and lower natural process limits

What Fig 4 shows is that the UK’s 2000 Maths score falls just outside the upper natural process limit, so even if the OECD hadn’t told us it was an anomalous result, we’d know that something different happened to the process in that year. You might think this is pretty obvious because there’s such a big difference between the 2000 score and all the others. But what if the score had been just a bit lower?  I put in some other numbers:

score  Xbar  Rbar UCLR UNPL LNPL
529 (actual) 500.4 10.25 33.52 527.67 473.13
520 498.6 8 26.16 519.88 477.32
510 496.6 5.5 17.99 511.23 481.97
500 494.6 3 9.81 502.58 486.62

Table 3: outcomes of alternative scores for year 2000

Table 3 shows if the score had been 520, it would still have been outside the natural process limits, but a score of 510 would have been within them.

pisa-fig-5 Fig 5: UK Maths scores showing upper and lower natural process limits for a year 2000 score of 510

ups, downs and targets

The ups and downs of test results are often viewed as more important than they really are; up two points good, down two points bad – even though a two-point fluctuation might be due to random variation.

The process control model has significant implications for target-setting too. Want to improve your score?  Then you need to work harder or smarter. Never mind the fact that students and teachers can work their socks off only to find that their performance is undermined by a crisis in recruiting maths teachers or a whole swathe of schools converting to academies. Working harder or smarter but ignoring natural variation supports what’s been called Ackoff’s proposition – that “almost every problem confronting our society is a result of the fact that our public policy makers are doing the wrong things and are trying to do them righter”.

To get tough on PISA scores we need to get tough on the causes of PISA scores.

 Reference

Wheeler, DJ (1993).  Understanding variation: The key to managing chaos.  SPC Press Inc, Knoxville, Tennessee.