apprentice without a sorcerer

Cummings’ essay Some Thoughts on Education and Political Priorities highlights his admiration for experts, notably scientists, but this doesn’t prevent him making several classic novice errors. These errors, not surprisingly, lead Cummings to some conclusions contradicted by evidence he hasn’t considered. I’ve focused on four of them.

oversimplifying systems

Cummings knows that systems operate differently at different levels, and although all systems, as part of the physical world involve maths and physics, you can’t reduce all systems to maths and physics (p.18). But his preoccupation with maths and physics, and lack of attention to the higher levels of systems suggest he can’t resist doing just that. In his essay maths is mentioned 473 times (almost 2 mentions per page) and physics 179 times. Science gets 507 references and quantum 238. In contrast, the arts get 8 mentions and humanities 16. Ironically, given his emphasis on complex systems, Cummings seems determined to view complex knowledge domains like education, politics, the humanities and arts, only through the lenses of maths, physics and linear scales.

Cummings’ first degree is in history, but he knows a lot of scientific facts. How deep his understanding goes is another matter. He opens the section on a scientific approach to teaching practice with the famous ‘Cargo Cult’ speech in which Richard Feynman accused educational and psychological studies of mimicking the surface features of science but not applying the deep structure of the scientific method (p.70). Cumming’s criticism is well-founded; evidence has always influenced educational practice in the UK, but the level of rigour involved has varied considerably. Ironically, Cummings’ appeal to scientific evidence then itself sets off down the cargo-cult route.

misunderstanding key concepts: chunking vs schemata

Cummings claims “experts do better because they ‘chunk’ together lots of individual things in higher level concepts – networks of abstractions – which have a lot of compressed information and allow them to make sense of new information (experts can also use their networks to piece together things they have forgotten)” (p.71).

‘Chunking’ occurs when several distinct items of information are perceived and processed as one item. The research e.g. Miller (1956), De Groot (1965) and Anderson (1996), shows it happens automatically after groups of low-level (simple) items with strongly similar features have been encountered very frequently, e.g. Morse code, words, faces, chess positions. I’ve not seen any research that shows the same phenomenon happening with information that’s associated but complex and dissimilar. And Cummings doesn’t cite any.

Information that’s complex and dissimilar but frequently encountered together (e.g. Periodic Table, biological taxonomy, battle of Hastings) forms strong associations cognitively that are configured into a schema. What Cummings describes isn’t chunking; it’s the formation of a high level schema. Chunks are schemata, but not all schemata are chunks.

Cummings is right that experts abstract information to form high level schemata, but the information isn’t compressed as he claims. The abstractions are key features of aspects of the schema e.g. key features of transition metals, birds or invasions.  I can just about hold all the key features of birds in my working memory at once, but not at the same time as exceptions (e.g ostrich, penguin) or features of different bird species. The prototypical features make it easier to retrieve associated information, but it isn’t retrieved all at once. If I think about the key features of birds, many facts about birds and their features spring to mind, but they do so sequentially, not at the same time. The limitations of working memory still apply.

The distinction between chunking and schema formation is important because schemata play a big part in expertise e.g. Schank & Abelson (1977) and Rumelhart (1980). Despite their importance, Cummings refers to schemata only once, when he’s describing how his essay is structured (p.7). The omission is a significant one with implications for Cumming’s model of how experts structure their knowledge.

experts vs novices

Experts in a particular field derive their expertise from a body of knowledge that’s been found to be valid and reliable. They construct that knowledge into schemata, or mental models. New knowledge can then be incorporated into the schemata, which might then need to be configured differently. Sometimes experts disagree strongly, not about the content of their schemata, but about how the content is configured.

The ensuing debates can go on for decades. A classic example is the debate between those who think correlations between intelligence test scores indicate that intelligence is a ‘something’ that ‘really exists’, and those who think the assumption that there’s a ‘something’ called intelligence, shapes the choice of items in intelligence tests, so correlations should come as no surprise (see previous post). Another long-standing debate involves those who think universal patterns in the structure of language mean that language is hard-wired in the brain, versus others who think the patterns emerge from the way networks of neurons compute information.

Acquiring key information about an unfamiliar knowledge domain takes time and effort, and Cummings has obviously put in the hours. What’s more challenging is finding out how domain experts configure their knowledge – experts often take their schemata for granted and don’t make them explicit. Sometimes you need to ask directly (or be told) why knowledge is organized in a certain way, and if there are any crucial differences of opinion in the field.

Cummings doesn’t seem to have asked how experts structure their knowledge. Instead, he appears to have squeezed knowledge new to him (e.g. chunking) into his own pre-existing schema without checking whether his schema is right or wrong. Or, he’s adopted the first schema he’s agreed with (e.g. genes and IQ). He admits to basing his genes/IQ model largely on Robert Plomin’s Behavioural Genetics and talks by Stephen Hsu. He dismisses the controversies and takes Plomin and Hsu’s models for granted.

evaluating evidence

There are references to the scientific method in Cummings’ essay but they’re about data analysis, not the scientific method as such. A crucial step in the scientific method is evaluating evidence – analysing data for sure, but also testing hypotheses by weighing up the evidence for and against. This process isn’t about ‘balance’ – it’s about finding flaws in methods and reasoning in order to avoid confirmation bias.

But Cummings repeatedly accepts evidence in support of one thing or against another, without questioning it. I’d suggest he can’t question much of it because he doesn’t know enough about the field. Some that caught my eye are:

  • Assuming hunter-gatherers’ knowledge is “based on superstition (almost total ignorance of complex systems)” (p.1). Anthropology that might claim otherwise, is like other social sciences, summarily dismissed by Cummings.
  • Unsubstantiated claims such as “Aeronautics was confined to qualitative stories (like Icarus) until the 1880s when people started making careful observations and experiments about the principles of flight” (p.21). Da Vinci, Bacon, Montgolfiers, Caley? No mention.
  • Attributing European economic development between 14th and 19th centuries to ‘markets and science’ and omitting the role of the Reformation, French Revolution, or Enclosure Acts (p.108).
  • Uncritical acceptance of Smith’s and Hayek’s speculative claims about the benefits of markets (p.106).
  • Overlooking systems constraints on growth – in corn yields, computing power etc. (pp.46, 231-2). No mention of the ubiquitous sigmoid curve.
  • Overlooking the Club of Rome’s Limits to Growth when discussing shortage and innovation (p.112).
  • Emphasising the importance of complex systems with no mention of systems theory as such (e.g. Bertalanffy’s general systems theory).
  • Ignoring important debates about construct validity e.g. intelligence and personality (p.49).

not just wrong

People are often wrong about things and usually a few minor errors don’t matter. In Cummings’ case they matter a great deal, partly because he’s so influential, but also because even tiny errors can have huge consequences. I chose the example of chunking because Cummings’ interpretation of it has been disproportionately influential in recent English education policy.

Daisy Christodoulou in Seven Myths about Education (2014) takes the assumption about chunking a step further. She’s right that chunking low-level associations such as times tables allows us to ‘cheat’ the limitations of working memory, but wrong to assume (like Cummings) high-level schemata do the same. And flat-out wrong to claim “we can summon up the information from long-term memory to working memory without imposing a cognitive load.” (Christodoulou p.19, my emphasis). Her own example (23,322 x 42) contradicts her claim.

Christodoulou’s claim is based on Kirschner, Sweller & Clark’s 2006 paper ‘Why minimal guidance during instruction does not work’. The authors say; “The limitations of working memory only apply to new, yet to be learned information that has not been stored in long-term memory. New information such as new combinations of numbers or letters can only be stored for brief periods with severe limitations on the amount of such information that can be dealt with. In contrast, when dealing with previously learned information stored in long-term memory, these limitations disappear.” (Kirschner et al p.77).  The only evidence they cite is a 1995 review paper proposing an additional cognitive mechanism “long-term working memory”.

I have yet to read a proponent of Kirschner, Sweller & Clarke’s model discuss the well-known limitations of long-term memory, summarised here. Greg Ashman for example, following on from a useful summary of schemata, says;

One way of thinking about the role of long-term memory in solving problems or dealing with new information is that entire schema can be brought readily into working memory and manipulated as a single element alongside any new elements that we need to process. The normal limits imposed on working memory fall away almost entirely when dealing with schemas retrieved from long-term memory – a key idea of cognitive load theory. This illustrates both the power of having robust schemas in long-term memory and the effortlessness of deploying them; an effortlessness that fools so many of us into neglecting the critical role long-term memory plays in learning”.

Many with expertise as varied as English, history, physics or politics, have enthusiastically embraced findings from cognitive science that could improve the effectiveness of teaching. Or more accurately, they’ve embraced Kirschner, Sweller and Clarke’s model of memory and learning.  Some of the ‘cog sci’ enthusiasts have gone further. They’ve taken a handful of facts out of context, squeezed them into their own pre-existing schemata, and drawn conclusions that are at odds with the research. They’ve also assumed that if an expert in ‘cog sci’ makes a plausible claim it must be true, but haven’t evaluated the evidence cited by the expert – because they don’t have the relevant expertise; cognitive science is a knowledge domain unfamiliar to them.

Nevertheless objections to the Kirschner, Sweller and Clarke model are often dismissed as originating either in ideology or ignorance. Ironic, as despite emphasising the importance of knowledge, evidence and expertise, many of the proponents of ‘cog sci’ are patently novices selecting evidence to support a model that doesn’t stand up to scrutiny. Murray Gell-Man is right that we need people who can take a crude look at the whole of knowledge (p.5), but the crude look should be one informed by a good grasp of the domains in question.

In 1797, Goethe published a poem entitled Der Zauberlehrling (Sorcerer’s Apprentice). It was a popular work, and became even more popular in 1940 when animated as part of Disney’s Fantasia, with Mickey Mouse playing the part of the apprentice who started something he couldn’t stop. The moral of the story is that a little knowledge can be a dangerous thing. Cummings has been portrayed as a brilliant eccentric and/or an evil genius. I think he’s an apprentice without a sorcerer.

references

Anderson, J (1996) ACT: A simple theory of complex cognition, American Psychologist, 51, 355-365.

Christodoulou, D (2014).  Seven Myths about Education.  Routledge.

de Groot, A D (1965).  Thought and Choice in Chess.  Mouton.

Kirschner, PA, Sweller, J & Clark, RE (2006). Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching Educational Psychologist, 41, 75-86.

Miller, G (1956). The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information, Psychological Review, 63, 81-97.

Rumelhart, DE (1980). Schemata: the building blocks of cognition. In R.J. Spiro et al. (eds) Theoretical Issues in Reading Comprehension.  Lawrence Erlbaum: Hillsdale, NJ.

Schank, RC & Abelson, RP (1977). Scripts, Plans, Goals and Understanding: an Inquiry into Human Knowledge Structures.  Lawrence Erlbaum: Hillsdale, NJ.

 

 

 

the dead sheep in the stream and new special needs legislation

Many years ago, on a walking holiday in the Lake District with friends, the conversation turned to how clean the water in the mountain streams might be. One of the more intrepid members of our party said; “So it would be OK for me to drink this?” “Probably,” replied an experienced fell-walker, “But not if there’s a dead sheep in the beck higher up.”

mountain stream

I was reminded of this incident by my local parent carer group newsletter. Not that there was anything wrong with the newsletter itself, but it included a couple of articles about the proposed legislative changes for the support of children with special educational needs and disabilities (SEND). The proposals include;

• joint planning and commissioning of services by local authorities and Clinical Commissioning Groups
• individual support specified in a single Education, Health and Care (EHC) plan
• support extended to age 25 and
• that families of children with EHC plans should have the option of a personal budget.

The proposals have, overall, been welcomed. However, concerns have also been expressed.

The changes were first put forward in March 2011 in a Green Paper entitled “Support and aspiration: A new approach to special educational needs and disability”. I was involved in the responses of several groups to the consultation that followed and the general feeling was that it was difficult to comment on the viability of the proposals because they hadn’t been set out in enough detail. The Department for Education’s response to the consultation, “Support and aspiration: A new approach to special educational needs and disability – progress and next steps” was published in May this year and draft legislation was published in September. The Department for Education appointed 20 pathfinder authorities to pilot and evaluate the proposed changes, with a final evaluation due in the summer of 2013 – almost a year later. Queries from interested parties about how the proposals would be implemented were generally greeted with advice to wait for the pathfinder reports. In the event, not surprisingly, the pathfinder evaluation has been extended and it is likely that the legislative programme will be delayed until after the final pathfinder evaluation is published.

In discussions about these changes, I’ve felt like a Cassandra, prophesying doom and gloom whilst many around me have remained relentlessly upbeat. After all, the fact that there’s a SEND Bill at all shows that the current government recognizes there are problems with the current system, and the proposed changes show that the DfE knows what the biggest ones are. Many children are likely to benefit from the changes. But in my view the proposals merely tweak problems caused by much more fundamental factors in the system, and that if these factors aren’t addressed, the current set of problems will simply be exchanged for another. One parent I sat next to in a meeting kept saying “At least it couldn’t be worse than the current system.” Well, actually it could. It could be like the situation prior to the Warnock report in 1978, which recognized that many able children were denied a suitable education because of a physical disability, and many less able children were considered ineducable. Or, as I suggested, the legislation might result in a set of problems that are simply different to the current ones.

Components of a service

A service – whatever it is and whoever it’s for – has to have several components. First, expertise. The people offering the service need to know how to accurately assess their clients’ needs and how best to meet them. Second, capacity and resources. An effective service will need enough people with the right expertise and sufficient equipment, materials, buildings etc. Thirdly, before designing the system the service will need to carry out a requirements analysis for all the people who need the service – usually described as a needs analysis in the case of children with SEND. No service would have an unlimited budget, so once planners and commissioners know what the needs are, they can then decide what expertise and resources are going to be most cost effective and what service users can probably manage without. This might seem self-evident and might appear to be what central and local government are doing already, but since the current system of support for children with SEND clearly isn’t working – and I would argue that it never has worked, in terms of ensuring that most children with SEND achieve their full potential – there must be something going wrong somewhere.

What’s going wrong?

The Department for Education seems to have decided that the problem lies in the way support services are planned, commissioned and delivered. Planning and commissioning aren’t joined up enough, despite local authorities having integrated children’s services for nearly a decade. The process of statutory assessment is too cumbersome and takes too long, even though in principle, assessments could be completed within weeks, rather than months. Support doesn’t go on for long enough, despite adult services being available. Local authorities aren’t allocating finance in the most effective way, even though it’s their job to do so. Consequently, the planning, commissioning and delivery of the system are being changed. Since the people who designed the current system presumably thought it would work, and viable processes for planning, commissioning and delivery are already in place, a key question does not appear to have been asked; what made the system go wrong in the first place?

The dead sheep in the stream

This is where the sheep in the stream analogy comes in. Imagine that you live in a farmhouse at the foot of a mountain. The farm is too remote for a mains water supply and for three hundred years the inhabitants have relied on water from a stream fed by a spring halfway up the mountainside. The purity of the water is renowned locally and the only problems ever reported have been that the stream flows sluggishly during extreme droughts. Then one day everyone at the farm gets sick. The illness is identified as a water-borne one and further investigation reveals the source – the body of a sheep lying in the stream just below the spring, hidden in a densely wooded area where sheep rarely stray. The farming family is advised to boil their drinking water or install a purification unit, but they might not need to do anything that involves that level of inconvenience or expense. It’s quite likely that simply removing the body of the sheep from the stream and letting the water flow for a couple of days would allow farmers to continue drinking the spring water for the next three hundred years without mishap – provided no more bodies end up in the stream.

Requiring local authorities to undertake joint planning and commissioning, implementing EHC plans, extending children’s services to 25 and providing personal budgets are all the equivalent of the farming family boiling their water in pots instead of kettles or installing a more sophisticated purification unit – while there’s still a dead sheep in the stream that’s contaminating the water. So what’s the equivalent of the sheep? I’d say it was a problem with each of the three components of service provision I mentioned earlier – expertise, capacity and resources, and requirements analysis – not downstream in the system near the point of delivery where most of the amendments are taking place, but further upstream.

Expertise

First, let’s look at expertise. Recent independent reports have indicated a lack of expertise with regard to children, in the education (Lamb, 2009), health (Kennedy, 2010) and social care (Munro, 2011) sectors. Despite the Warnock recommendation that children with SEND be taught in mainstream schools where possible being implemented since 1978, it’s only since 2009 that teachers have been required to have SEN training and that new special educational needs co-ordinators (SENCOs) have had to be qualified teachers. Teaching Assistants (TAs), who now make up around 25% of the mainstream school workforce, are generally not qualified teachers and don’t necessarily have any educational training, but are often the people who spend most time with children with SEN. A recent study (Webster & Blatchford, 2012) revealed that teachers aren’t usually trained to work with TAs, so many TAs are having to work ‘on the hoof’ in the classroom with little or no preparation with a child with learning or behavioural problems. The study found that when TAs worked with the rest of the class for part of the lesson so teachers could spend time with the children with SEN, the achievement of the pupils improved and teachers understood their learning difficulties better. What’s puzzling is how this situation arose in the first place. Here’s an extract from a piece about SENCO training published in the Times Educational Supplement in May 2009.

The [training] courses have been set up to address serious concerns about the perceived “low status” of Sencos and to raise the profile of special needs and disabilities in schools.”

I find it intriguing that although the professional status of SENCOs and poor awareness of special educational needs might be relevant issues, the TES reporter frames SENCO training in those terms of rather than in terms of the expertise required to help all children learn. What does this say about perceptions of SEN?

Capacity and resources

A second factor is capacity and resources; I’ll talk about capacity first. A recurring problem for parents of children with SEND is how long it takes to see professionals who can carry out assessments. Often all children get is repeated assessments; because of limited service capacity sometimes parents (and occasionally teachers) are expected to implement therapies even though they have no idea what might be causing the child’s problems or what outcomes to expect. Another recent report (Bercow, 2008) suggested that speech and language therapy in England was a postcode lottery, and there doesn’t seem to have been a significant improvement since then. The British Psychological Society has expressed concerns (not for the first time) about cuts in the number of educational psychologists employed by local authorities. Google ‘shortage occupational therapist’; and you’ll find reports from various parts of the globe. Then there’s resources. Parents report problems getting wheelchairs and nappies; even the NHS website says that there might be a waiting list for assessments (waits for the actual wheelchair aren’t even mentioned). My local occupational therapy service apologized for the delay in providing therapy for my son. One problem was that they hadn’t been able to access his school to show teachers how to integrate exercises into his school day. Another obstacle was that because their equipment takes an hour to put up and an hour to dismantle, the only time they were able to book a room large enough and available for long enough for them to treat several children in one day was during the school summer holidays.

Requirements analysis

And then there’s the requirements analysis. Under the 1989 Children Act, local authorities are required to keep a register of children with disabilities. This should provide the information they need to enable them to design support services. The register is a voluntary one in the sense that parents volunteer information about their children, and there are obviously questions over what qualifies as a disability, so at best such a register is only going to provide approximate information about the needs of children with disabilities in a given locality. But an approximation is all that’s required. In the past twenty years, it should have been possible to form a fairly accurate picture of local needs, trends over time and year-to-year fluctuations. But judging by recent reports, support for children with SEND has been getting worse, rather than better. So what’s gone wrong?

I suggest that because education, health and social care systems have been evolving piecemeal during this time, national government initiatives have cut across local authorities’ ability to use data to design effective services. For example, following the Warnock report in 1978, local authorities were encouraged to educate children with disabilities in mainstream schools where possible. An inspiring example of this is the collaboration between a mainstream junior school and a school for children with visual impairment described by Hegarty and Pocklington (1981). At that time, local authorities and individual schools had complete control over such initiatives. Then in 1988, the Education Reform Act introduced a compulsory national curriculum, followed in 1991 by national curriculum assessments, commonly known as SATs. Although there might have been good reasons for introducing both, they have each had an impact on the Warnock recommendation for the inclusion of SEND pupils in mainstream schools. If the performance of schools is assessed by pupils’ performance in standardized tests, systems pressures will inevitably lead to a tendency to marginalize pupils with SEND, either overtly – by schools discouraging admittance or by formal or informal exclusions – or covertly by simply not allocating sufficient resources to their education. Add to this the absence of SEN from initial teacher training and the reduction in SEN expertise within the education system as a whole due to a focus on children within the normal range and the closure of special schools, and no amount of tinkering with statutory assessments or who holds budgets will be able to compensate.

Failure demand

Overlooking shortcomings in factors that are upstream in a system means that whatever you do to problems downstream, they won’t get fixed. In fact the upstream issues create the need for further resources that wouldn’t be needed if the upstream problems were fixed. This phenomenon is what John Seddon calls failure demand – demand created solely by failures of system design. A common failure demand in the case of children with SEND is that avoiding early intervention in an attempt to avoid unnecessary costs often means that simple problems become complex ones, requiring expensive interventions later on. Not to mention the sometimes permanent damage done to a child’s self-esteem and the time wasted by teachers, parents and professionals trying to get problems resolved in the meantime. Providing sufficient resources to meet needs might not cost more; in fact, once failure demand is eliminated, costs can go down.

In short, until teachers, healthcare and social care professionals are trained to meet the needs of all children, not just those within the normal range, until there are enough people with that training working within the education, health and social care sectors, and until there are enough materials, equipment and space available to meet the needs of all children, the needs of all children will not, and cannot be met.

References

Hegarty S. and Pocklington K. (1981). “A junior school resource area for the visually impaired” in W. Swann (Ed.) The Practice of Special Education, Basil Blackwell/Open University Press.

Webster R. & Blatchford P. (2012). “Supporting learning?:.How effective are teaching assistants?” in P. Adey & J. Dillon (Eds) Bad Education: Debunking myths in education, McGraw Hill.

Acknowledgements

Photograph: Tullynaglack, Donegal, copyright Louise Price, used under Creative Commons http://commons.wikimedia.org/wiki/File:Mountain_stream,_Tullynaglack_-_geograph.org.uk_-_974248.jpg