standardised testing: what’s it good for?

A campaign by parents to keep their children off school on Tuesday 3rd May as a protest against SATs prompted a Twitter discussion about the pros and cons of standardised tests. One teacher claimed that they’re important because they hold schools to account. I think that’s a misuse of standardised tests. First, because test results are a poor proxy measure of teaching quality. Second, good teaching (and hard work on the part of the student) are necessary but not sufficient conditions for good test performance. Third, using test results to hold schools to account overlooks the natural variation inherent in large populations.

test results as a measure of teaching quality

Tests such as the National Curriculum Tests (commonly known as SATs) GCSEs and A levels sample students’ recall and understanding of a particular body of knowledge – the KS2 curriculum, GCSE/A level course. The knowledge is sampled because testing the student’s knowledge of all the material in the course would be very time consuming and unwieldy. In other words, test results are a proxy for the student’s knowledge of the course material.

But the course material itself is a proxy for all that’s known about a particular topic. KS2 students learn basic principles about how atoms and molecules behave, GCSE and A level students learn about atomic theory in more detail, but Chemistry undergraduates complain that they have to then unlearn much of what they were taught earlier because it was the simplified version.  So test results are actually a second order proxy for the student’s knowledge of a particular topic.

Then factors other than the student’s knowledge impact on test results. The student might be unwell on the day of the test, or might have slept badly the night before. In the months before the test they might have been absent from school for weeks with glandular fever or their parents might have split up. In other words, test results are affected by factors other than teaching and learning; factors beyond the control of either the school or the student.  In other words, test results are a weak proxy for both the quality of teaching and the student’s knowledge.

good teaching and hard work are necessary but not sufficient for good test performance

There’s an asymmetry between the causes of high and low test results. It’s difficult to get a high test score without hard work on the part of the student and good teaching on the part of the school.   But there are many reasons why a student might get a low score despite hard work and good teaching.

That’s at the individual level. Similarly at the school level it’s safe to conclude that a school with consistently good results in national tests is doing its job properly, but it’s not safe to conclude that a school that doesn’t get consistently good results isn’t.

The education system has been plagued over the years by two false assumptions about student potential. Either that all students have the potential to get good test scores and that good teaching is the key determining factor, or that students from certain demographic groups won’t get good test scores however well they’re taught. In reality it’s more complicated than that, of course. Students from leafy suburbs are more likely to do well in tests for many reasons; even if they are taught badly, they have access to resources that can sometimes compensate for that. Students from the kind of housing estate that motivates Iain Duncan Smith are at a higher risk of adverse life events scuppering their chances of getting good test results no matter how good the teaching at their school. And the older they get, the more adverse life events they are likely to encounter.

So, test results are a pretty good first order proxy for a student’s knowledge of course material. They are a not-so-good second order proxy for a student’s knowledge of the topic the course material represents. And only a weak proxy for quality of teaching.

life is just one damn thing after another*

Those in favour of standardised testing often cite cases of particular schools in deprived areas that have achieved amazing outcomes against the odds. Every child can read by the age of six, or is fluent in French, or whatever.   The implication is that if one school can do it, all schools can. In principle, that’s true. In principle, all head teachers can be visionaries, all teachers can be excellent and all families can buy in to what the school wants to achieve.

But in practice life doesn’t work like that. Head teachers get sick, senior staff have to work part-time because of family commitments, local housing is unaffordable making recruitment a nightmare, or for many families school is just one more thing they can’t quite keep up with.

On top of that, human beings are biological organisms. Like all populations of biological organisms we show considerable variation due to our genes, our environment and interactions between the two. It might be possible to improve test performance across the education system, but there are limits to the improvement that’s possible. Clean water and good sanitation increase life expectancy, but life expectancy doesn’t go on increasing indefinitely once communities have access to clean water and sanitation. Expecting more than 50% of children in primary schools to perform above average simply shows a poor grasp of natural variation – and statistics.

standardised testing: what is it good for?

Standardised testing in primary schools makes sense. It samples children’s knowledge of key material. It allows schools to benchmark attainment. Standardised testing as a performance measure can alert schools to problems that are impacting on children’s learning.

However, the reasons for differences in students’ performance in standardised tests are many and varied. Performance will not improve unless the reasons for poor performance are addressed. Sometimes those reasons are complex and not within the schools’ remit. To address them local families might need better public services, better jobs or better housing – arguably not the core responsibility of a school. Poor teaching might not be involved at all.

However, successive governments haven’t used test results simply as broad indicators of whether a school is on track or whether there are problems that need to be addressed (not necessarily by the school), but as a proxy for teaching quality.  Test results have been used to set performance targets and determine funding, regardless of whether schools can control the factors involved.

This shows a poor understanding of performance management§, and it’s hardly surprising that the huge amounts of money and incessant policy changes thrown at the education system over recent decades have had little impact on the quality of education of the population as a whole.

Notes

*A quotation attributed to Elbert Hubbard, an American  writer who died when the Lusitania was sunk in 1915.

§ The best book I’ve read on performance management is a slim volume by Donald Wheeler called Understanding variation: The key to managing chaos.  A clearly written, step-by-step guide to figuring out if the variation you’ve spotted is within natural limits or not.  Lots of references to things like iron smelting and lumber yards, but still very relevant to schools.

 

 

 

 

how to run a government – or not

A decade ago, the then government heard the cries of overworked teachers and implemented an initiative intended to reduce their workload. Statutory planning, preparation and assessment (PPA) time was introduced; teachers were to be freed up from at least 10% of their contact time with students. My children’s primary school sent a letter home about it. Cover was to be provided by teaching assistants.

Since time immemorial, teachers’ working days have been relatively short- if somewhat intense – and they’ve had long vacations (not ‘holidays’, note), giving them ample time for PPA. So why the sudden need to resort to statutory guidance in order to give teachers time to carry out an integral part of their job?

The answer, of course, is that since the Education Reform Act 1988 an entire performance industry had sprouted from the education system, and teachers spent so much time servicing its bureaucratic demands they didn’t have time to do their jobs. But the reasons why PPA time needed safeguarding didn’t appear to have been considered by government.

Baffled, and concerned about both the need for and consequences of this initiative, I wrote to the school, our LA and the Department for Education and Skills (DES). Rather to my surprise, I got replies from each of them, the gist of which was along these lines:

School: *Sigh*. Yes, quite, but we’ll do our best (with yet another initiative that requires yet more reorganisation).

LA: (Phone call from primary HT seconded to LA.) You’ve summed up our concerns exactly. Do we have permission to quote you?

DES: (A month later). We are required to reply to your letter within 20 working days. This constitutes a reply. We’ll respond to the points you made as soon as we can.
(Another month later). We are required to reply to your letter within 20 working days. This constitutes a reply. We’ll respond to your concerns as soon as we can.
(Another month later). We’ve addressed the problem of teachers’ workload. If it doesn’t work, it’s the schools’ and LAs’ fault.

I also wrote to Boots’ Health & Beauty Magazine, which had featured an article about an over-worked teacher. It recommended products she could use to help her sleep, relax, boost her immune system and look less tired. There was no mention of the possibility of reducing her working hours. Their response? Essentially “teachers’ working hours are nothing to do with us”.

Taken at face value, the statutory guidance was successful. Teachers did get additional PPA time. But ten years on it doesn’t appear to have made a significant dent in their workload. That’s because PPA time itself wasn’t the cause of the problem.

How to Run a Government so that Citizens Benefit and Taxpayers Don’t Go Crazy

The impact (positive and negative) of the PPA initiative illustrates the flaw at the heart of Sir Michael Barber’s ‘delivery science’ (formerly known as ‘deliverology’) described in detail in his latest book, How to Run a Government so that Citizens Benefit and Taxpayers Don’t Go Crazy. Barber, a former history teacher, Hackney councillor, NUT official and education policy adviser, was head of Tony Blair’s Prime Minister’s Delivery Unit (PMDU) from 2001-2005. He’s since worked for McKinsey and is currently chief education adviser to educational publishers Pearson.

The Delivery Unit’s job was to ensure that policies were actually implemented. Part of the Prime Ministerial role is to co-ordinate government so having a team dedicated to tracking the progress of particular policies and removing obstacles from their path makes sense. In How to Run a Government, Barber discusses prioritisation, organisation, strategy, planning, routines, problem-solving, ensuring that policies are well-established and that the outcomes are what the electorate wants. He describes the tools and processes the PMDU used to get the job done. On the face of it, his book could be viewed as a practical handbook for anyone who, with little experience, has been tasked with chasing up the progress of particular policies. Or as a project management primer that doesn’t refer directly to the substantial existing literature on the subject. It also offers intriguing insights into how the Blair government functioned.

But scratch the surface and underneath you find something rather different. Barber calls his book How to Run a Government but it’s actually about how to run a Delivery Unit – not quite the same thing. Barber appears to think that the narrow focus required of his Delivery Unit means it’s OK to adopt an equally narrow frame of reference. And for Barber a government and a Delivery Unit amount to the same thing because he attributes inefficiencies and ineffectiveness in modern governments to the failure to implement promised policies. He twice cites Margaret Thatcher’s famous exasperated; “Don’t tell me what. I know what. Tell me how”. For Delivery Units the ‘what’ might be secondary. But for government it’s of primary importance.

what governments do

Traditionally, the focus of central government has been national security; the defence of the realm and the maintenance of law and order. But in recent years governments have gradually taken on what were once local responsibilities – utilities, transport, education, health and welfare. And governments are not just making sure the services are functioning and are properly resourced, they are attempting to manage them. There are good reasons why they shouldn’t. One is that government ministers or civil servants are unlikely to have the necessary specialist domain knowledge. Another is the big risk of services being used for political ends rather than for the benefit of the people. Nonetheless, rightly or wrongly, modern government involves co-ordinating the functions of several complex interconnected systems.

Complex interconnected systems are challenging things. Because they are complex and interconnected, what looks like a minor tweak to one bit can have a massive – sometimes catastrophic – impact on another. For example, removing ‘spare’ hospital beds, increasing GPs’ salaries and reducing social care budgets – all measures that in isolation seem perfectly reasonable – have resulted in the current A&E crisis. It’s imperative that people in government have a good understanding of how systems work. Barber’s book shows that they often don’t. Take for example, the origin of government policies.

where policies come from; a systems perspective

In his book The Open Society and Its Enemies Karl Popper suggests that the focus of a democratic government should be on resolving problems; he illustrates the point by highlighting the disasters that have resulted from attempts to instantiate utopian visions. Suppose for a moment that the next UK government agrees with Popper about its problem-solving role. Suppose also that it has accurately identified the most serious problems besetting the nation.

From a systems perspective, the next task would be to check out the causes of those problems. Causes can be complex, but it’s crucial to pinpoint them accurately in order to get the problems sorted. Sometimes a relatively inexpensive tweak to one system can lead to major improvements in several others – effective sanitation and access to clean water lead to big improvements in health and economic prosperity, for example. The specific policies would consist of whatever action was needed in order to address the causes of the problems. Before implementing the policies, government would need to pilot them to check out their actual impact. This is an essential phase because the complexity of system interactions makes it very difficult to predict the exact outcomes of any given change.

In other words, specific policies would emerge from an analysis of the systems within which the problems are embedded. Policies would be evidence-based and their consequences would be fairly predictable. Pilots would allow unintended or unwanted outcomes to be addressed before implementation.

But that’s not how Sir Michael sees policies. For him, they lie ‘somewhere’ between strategy and implementation (p.101). It doesn’t matter if they originate in ideology (p.62) or in the personal preferences of the PM or President (p.182). Barber recognises that policies can have unintended or unwanted outcomes, but attributes those to choosing the wrong targets (e.g. ones beyond the control of the PM – p.11) or to a failure of a target to ‘tell a good story’ or have sufficient ‘moral purpose’ (p.24). And for Barber, pilots and evidence-based policymaking represent an over-cautious approach (p.7).

One advantage of adopting a narrow frame of reference is that you can just focus on getting the job done. Another advantage is that you can overlook the problems that might emerge as a consequence.

looking past the problems

Barber talks with approval about Calvin Coolidge’s use of routine to implement budget cuts (p.164) but refers only in passing to ‘America’ (never mind anywhere else) being ‘plunged into depression’ even though the 1929 economic crash occurred shortly after Coolidge’s term of office ended and was arguably caused by factors he failed to address. Barber mentions the Iraq war only in passing too, in passages that deal in detail with a 5-hour meeting with Barber about asylum seekers (p.182) and Blair’s political capital leaking away (p.209).

In short, Barber’s narrow focus and narrow frame of reference allow him to get on with the job regardless of the consequences. Impressive results for delivery science; maybe not so impressive for the rest of us.

the emerging science of delivery

Barber claims How to Run a Government is about the ‘emerging science’ of delivery (p.xvii). To be sure he sets out 57(!) rules of ‘delivery science’ throughout the text – also gathered together in a convenient appendix. There’s a nine-page bibliography. And almost 300 pages on what Barber has learned about policy implementation. But that doesn’t make delivery a new ‘emerging science’.

Barber thinks it’s new because there are ‘countless books and manuals’ on various aspects of government, “but on how to get things done in government there is almost nothing. No manuals. Virtually no academic literature.” He goes on to say “Surveying the academic literature on the subject of political science…” (p.xvii). Maybe the clue to the missing manuals lies in where Barber looked for them. Maybe what politicians need to know about policy implementation isn’t filed under ‘political science’ because it’s already well-established in other domains.

Much of Barber’s bibliography is drawn from the literature on management; he doesn’t draw on (highly relevant) project management research; and organisational theory dating back at least to Weber (if not to Thucydides) and systems theory (tried and tested across multiple domains) don’t get a mention. In short, delivery isn’t an emerging science at all. What governments need to know about policy development and implementation is already out there. It just needs to be applied. The fact that recent governments have failed to apply it is worrying.

In his introduction, Barber quotes Charles I, who made a bit of a hash of his period as monarch; “There’s more to the doing than bidding it be done”*. There is indeed, but I’m not convinced Michael Barber has realised quite how much more there is.

Bibliography
Barber, M (2008). Instruction to Deliver: Fighting to Transform Britain’s Public Services. Methuen.
Barber, M (2015). How to Run a Government so that Citizens Benefit and Taxpayers Don’t Go Crazy. Allen Lane.
Popper, K (1945/2003). The Open Society and Its Enemies Vol 1: The Spell of Plato. Routledge.

* The source for this quote is cited as Barber’s Instruction to Deliver (2008). In it, the quote is cited without a source. A Google search appears to attribute it to a 2013 RSA lecture by Stein Ringen, mentioned in How to Run a Government. https://twitter.com/hashtag/RSARingen?src=hash I haven’t watched the Ringen lecture yet, but wouldn’t be surprised if he turns out to cite Barber. Did Charles I actually say it? Maybe we’ll never know.

brutal – and unworkable

This morning @HeatherBellaF posted an example on Twitter of what the principal of a high profile ‘flagship’ academy called her “direct style of management”. @HeatherBellaF’s comment was “Flippin’ heck! Brutal!”

In her first week at the academy, the principal says she asked the senior leadership team to “list those who they thought were inadequate teachers”. The names of almost a third of the 60+ teachers at the school came up. (We’re not told if the list included any of the senior leadership team.) A frank interview with each member of staff on the list followed. Within a year almost all had left.

Tales of ‘new brooms’ are a popular feature of management periodicals. Typically they mention the unsentimental removal of ‘dead wood’, the introduction of ‘new blood’ and the organisation in question being ‘turned around’ as a consequence. Such strategies are sometimes necessary, but most chief executives of private companies the size of the academy in question would think long and hard before replacing a third of their workforce in the space of a few months.

Private companies have responsibilities towards several groups of people. Shareholders, who provide capital; customers, the main source of income; their workforce, who enable them to operate; and taxpayers, who provide the infrastructure the company needs to function. All those groups of people need to be kept happy, or things go badly wrong. If a lot of shareholders sell their shares the company could go under; if customers go elsewhere, income dries up; if there’s high staff turnover productivity plummets; and if the firm avoids paying taxes the national infrastructure can suffer – and then everybody suffers.

For a private company, the strategy adopted by the academy principal would be a risky one. Leaving aside issues like whether senior managers would comply, the impact on the remaining staff or the union’s view, the outcome is unpredictable. In a best case, all those listed might see the error of their ways and pull up their socks and their performance. At the other extreme, all the staff in question might exit ASAP. That could be a problem for a medium-sized private company, because replacing staff costs money and time. Productivity and cashflow could suffer during those few months and if the company was struggling financially anyway, it could be the last straw. So most managers would adopt this approach only as a last resort, or would make sure the company could survive a few turbulent months before grasping the nettle.

Traditionally, state schools haven’t had to worry about the impact of recruitment costs on cashflow because they’ve had LEAs to fall back on, though questions would have been asked about a secondary school with a 30% staff turnover. But for a showcase academy, stumping up the cash to make sure teachers are top-notch and on message isn’t only good for the kids, it’s a sound marketing strategy.

The market economy model adopted by successive UK governments assumes that competition is a good thing because it fosters excellence. What this model conveniently overlooks is that the private sector looks healthy because we see only the companies that survive; not the 50% that don’t make it through the first three years. It also overlooks the fact that private companies can do whatever it takes to make a (legal) profit; if a particular type of customer isn’t profitable, the company can change its target demographic – unlike public education, health and social care that have to provide services for everybody.

It’s possible that all the teachers deemed ‘inadequate’ were simply not suited to that particular challenging academy and that the principal was right when she calmed her ‘moral qualms’ by hoping that the teachers who moved on would “succeed in a smaller, more orderly school”. That’s possible, but what if every ‘challenging’ school did the same? And what about the teachers who left having been told that “you (sic) have been identified by the leadership team as inadequate”?

Teacher training, recruitment and professional development are funded by all of us and it’s in all our interests to ensure that that investment is a productive one. The strategy of replacing teachers who are not up to the task might benefit a particular school, but the cost is borne by the community. So is the cost of another school recruiting them. So are the costs of teachers leaving the profession because having heard that they, personally, have been deemed inadequate has prompted them to think that making the ‘right decision about their career’ is to get out of teaching.

Many private companies, given the opportunity to replace ‘inadequate’ staff at no cost, would do so in a heartbeat because that would likely result in a rapid, substantial improvement in performance. So although the academy’s human resources management approach was effective, it was effective only because the costs were passed on to others, notably taxpayers. And the strategy won’t scale up; it will be effective for a few early adopters, but the pool of superlative teachers willing and able to work in a challenging school and have frank conversations with the principal isn’t a bottomless one.

It might have been more cost-effective in the long-term to have put in place a robust training programme that enabled the ‘inadequate’ teachers to become ‘adequate’. Or even ‘outstanding’.

This approach to teacher recruitment and retention isn’t just brutal – it’s unworkable.

help: NSPCC briefing on home education

The NSPCC briefing Home education: learning from case reviews highlights recommendations from seven serious case reviews (SCRs) published between 2008 and 2013 involving home-educated children. [The full briefing has since been replaced with a summary, but the original is still accessible here. Also note that the Serious Case Review for Child S listed in the NSPCC summary is for the wrong Child S.]

In the previous post I mentioned that the primary purpose of legislation is to protect the liberty of the individual. Historically the primary purpose of national government has been to protect liberty by defending the nation from attack from abroad, and of local government to do so by maintaining law and order.

But you’re unlikely to enjoy your liberty very much if you’re starving, sick or homeless. The massive increase in urban populations following the industrial revolution eventually resulted in the UK government, national and local, turning its attention to people’s quality of life. Over the last century or so national education, health and social care systems have been developed. Currently, education and healthcare are universal services, available to all. Significantly, social care isn’t.

social care for children and families

The parent of any child ‘in need’ according to the criteria set out in s.17 of the Children Act 1989 can request a social care assessment. The definition of a child ‘in need’ can be summarised as;

• unlikely to achieve or maintain a reasonable standard of health or development without the provision of services by a local authority
• health or development is likely to be significantly impaired, or further impaired, without the provision of such services; or
• disabled.

But there appear to be a number of hoops to jump through before a child ‘in need’ can hope to access support.

The first hoop is meeting the eligibility criteria for an initial assessment. Here, for example, are those of the Royal Borough of Greenwich. I’m not knocking the Royal Borough. Their criteria are explicit and specific. At least you know where you stand.

The second hoop is an initial assessment that determines whether or not the child is sufficiently ‘in need’ to be eligible for a core assessment. For many local authorities, the criterion for a core assessment is not so much about the child’s needs as about whether they are at risk.

The third hoop, the core assessment, presumably identifies what sort of support the child needs. Or not.

Despite s.17 of the Children Act 1989 saying

It shall be the general duty of every local authority…
(a) to safeguard and promote the welfare of children within their area who are in need; and
(b) so far as is consistent with that duty, to promote the upbringing of such children by their families,
by providing a range and level of services appropriate to those children’s needs.

the current focus of social care services appears to be on the safeguarding element of their general duties, not on the promotion of welfare element. The initial assessment prioritises safeguarding children already ‘in need’, not on preventing them from becoming ‘in need’ in the first place. This looks rather like locking the stable door after the horse has bolted.

In practice, it means you could be the parent of three children, each with complex disabilities and Statements of Special Educational Needs and repeatedly excluded from school, have a broken marriage and be estranged from your extended family as a consequence, be seriously sleep deprived, suffering from a bunch of mental and physical health problems and be unable to hold down a job because of your caring responsibilities, but if your children are not deemed to be at risk of harm or of harming others, you can still not be eligible for a core assessment.

Local authorities justify rationing support in this way by saying they must prioritise children who are most in need. On the face of it, this looks like a responsible use of taxpayers’ money. It isn’t of course. Low-level social problems don’t just resolve themselves because services are rationed; the costs are simply shifted elsewhere. Parents and children absorb many of the costs – financial and in terms of quality of life. Schools are expected to patch together the children’s lives and parents and children pitch up at GP practices with recurrent health problems. And in many cases social services end up having to support families anyway once their problems have escalated to the point where the children are deemed to be at risk of harm.

safeguarding

The task of safeguarding as presented in the seven SCRs involving home-educated children has three main features;

• identifying children already at risk
• monitoring their welfare
• intervening before they can come to harm

The focus of politicians, the media and the SCRs themselves has been on the failure to intervene in time to stop the children being harmed. The focus of those advocating increased regulation of home education is on monitoring the children’s welfare. The focus of social care assessments is on identifying children already at risk. It’s clear from the SCRs that none of the three features provide any guarantee that a child will stay safe – almost 30% of all children who were the subject of a SCR have had current or discontinued child protection plans. So why the focus on identifying, monitoring and intervention?

promoting welfare

The purpose of the seven SCRs was to examine the specific circumstances that led to a child coming to serious harm, not to catalogue all the support families received, so there’s little mention of what help parents asked for and what help they got. Most home educating families don’t want or need help, but in all seven cases cited by the NSPCC briefing parents had approached the authorities voluntarily at some point. That might simply have been impression management on their part, but let’s assume for the moment the approach was a request for help. What did they ask for? The evidence suggests that in six cases it involved specialist expertise; about developmental disorders, complex health problems, special educational needs, mental health and adoption. Specialist expertise isn’t always easy to access. Parents with children not deemed to be at risk have long reported difficulties getting it.

But not all the help needed was so highly specialised. In two cases, there were problems with housing. Child 3’s mother had sold all her furniture in an attempt to avoid the house being repossessed; it was during repossession that her daughter’s body was discovered. The chaos and tragedy into which Child 4’s family was precipitated appears to have been triggered by the mother’s attempts to improve their circumstances; she went on a diet, her partner moved in and she planned to relocate the family to a better neighbourhood. But her ‘healthy eating’ regime deprived her children of food, her partner had a troubled history, and she couldn’t get a tenancy transfer until she had ‘tidied up’ the house – the children had scribbled on the walls. Mrs Justice King, hearing the application for a care order for the surviving children, noted the chaos that the family descended into as a result of the mother’s partner attempting to lay laminate flooring throughout the house. Without wishing to trivialise the tragedy that followed, the outcome might have been very different if a lone parent with six children, some of whom had significant learning difficulties, had had access to old-fashioned services like meals on wheels and home help.

In some cases, the help the families did get appears to have made things worse. One parent experienced frustrating delays getting responses from the EOTAS service. Health interventions included two cases of over-prescription of medication, no conclusive diagnosis, being removed from a GP list and a suspicion of Fabricated or Induced Illness. Parents did get appointments with professionals – 130 in a year in one case, but the focus of social care appeared to be on assessing the risk to the children, rather than providing the support the family needed to lessen the risk. This might go some way to explaining the ‘combative’ and ‘aggressive’ attitude of parents.

Social workers, understandably, point out that a focus on high profile cases where they have failed to prevent children coming to harm doesn’t take into account the thousands of children that they help. That’s undoubtedly true, although I’ve never heard social workers elaborating on what sort of help families get; a recent interview on Radio 4’s Woman’s Hour (from 33m 40s) is a case in point. But I don’t think it’s the social workers who are responsible for the difficulty families have in getting help; the way social care services are designed makes this situation inevitable.

it’s the system

Most of us, at some times in our life, are faced with challenges we can’t cope with on our own. Some people have robust social networks they can call on for help. For others, their social networks are part of the problem. That’s what public sector services are for; to help people cope with those challenges.

In the early 1900s, the first social worker training course offered by the University of Birmingham included topics like British constitution, industrial history, economic analysis, statistics, law, and sanitation and hygiene. The emphasis was on understanding the wider context for the problems families faced. In the intervening decades, the focus has shifted from designing systems that improve people’s lives, to resolving individuals’ specific problems, to, more recently, a preoccupation with procedures.

For example, the 2013 statutory guidance Working Together to Safeguard Children reads less like statutory guidance than a handbook consisting of 97 pages of advice. Some of the advice is probably useful – there are flow charts for different types of cases, for example. But is it really necessary to tell experienced professionals planning a strategy discussion that it “might take the form of a multi-agency meeting or phone calls and more than one discussion may be necessary” (p.33)? It’s not surprising, with this level of micromanagement and the dire consequences of not paying attention to it, that local authorities’ focus is on following the procedures related to safeguarding rather than on promoting children’s welfare.

failure demand

If people don’t get the help they need when they need it, their problems don’t just go away. Services then have to deal with repeat referrals, complaints and worsening problems. This is what John Seddon calls ‘failure demand’ – demand caused solely by a failure to do the job properly in the first place. Successive governments have believed that the obvious way to deal with shortcomings is to use carrots and sticks to get services back on the strait and narrow. Unfortunately, the obvious way isn’t always the most effective. The focus of those working in the public sector has gradually shifted away from serving the public towards collecting the carrots and avoiding the sticks.

Increasing regulation in response to service failures, which is how government has responded and what the SCRs and the NSPCC recommend, won’t help. It will simply add to the services’ workload and make it even more difficult for them to help the people who need it. Counter-intuitively, what would reduce demand and service failures is making social care universally accessible and ensuring that front line professionals have the expertise, resources and flexibility to help prevent minor problems escalating into major ones. Not only is that likely to reduce overall demand but it could also make people’s lives better.

the new SEN legislation and the Dunkirk spirit

In less than a week an event will take place that’s been awaited with excitement, apprehension, or in some cases with something approaching the Dunkirk spirit. On 1 September part 3 of the Children and Families Act 2014 comes into force. It’s been described as the biggest change to special educational needs in 30 years.

It won’t work
. If I were a betting sort of person, I’d put money on the next government having to review the system again in a couple of years. How can I be so sure? Or so pessimistic? It’s because the ‘problem’ with special educational needs and disabilities (SEND) isn’t the special educational needs and disabilities, it’s the education system. And not just the SEN bit of it – it’s the education system as a whole. To find out why we need to go back in time…

we have a history

Education became compulsory in England in 1870. The new education system was essentially a one-size-fits-all affair focusing on reading, writing and arithmetic. Or more accurately one-size-fits-most; what took the government by surprise was the number of children turning up to school who didn’t fit the education system. Government essentially saw these ‘handicapped’ children as a problem, and its solution was to provide special schools for them. Although the solution made perfect sense, it wasn’t entirely successful. Handicapped children often ended up socially marginalised and sometimes institutionalised, and there were still children in mainstream schools who were struggling.

By the 1970s, the education system had changed considerably. There was more emphasis on an individualised education and local education authorities (LEAs), schools and teachers had a good deal of flexibility in the education they provided. The time was right for Margaret Thatcher as Secretary of State for Education to commission a review of the education of handicapped children, headed by Mary Warnock. The Warnock Committee reported in 1978. It defined special education as ‘provision not generally available in normal schools’ (p.45). In other words it saw the ‘problem’ of special education not as the children but as the educational provision available in mainstream schools. The committee’s recommendations fed into the 1981 Education Act that:

• assumed children would attend mainstream schools where possible
• did away with the old categories of handicap
• introduced the concept of ‘special educational needs’
• gave LEAs a duty to assess children’s special educational needs and to fund the additional provision required for their education.

The Act had the potential to transform the lives of children marginalised by the education system, but it clearly hasn’t done so – not in a good way, anyway. In the last 20 years we’ve had three SEN Codes of Practice, numerous inquiries, reports and tinkerings with SEN legislation and regulations. One select committee described the system as not fit for purpose. So…

what went wrong?

The Warnock recommendations were made in the context of a highly flexible education system. A contemporary account describes a fruitful collaboration between a school for children with visual impairment (VI) and a mainstream junior school, pioneered by a keen LEA officer (Hegarty & Pocklington, 1981). Children with VI were gradually integrated into the mainstream school and teachers trained each other. Everybody won.

In order to undertake such a project, LEAs, schools and teachers needed a fair amount of control over their time and budgets. Projects like this might have eventually been rolled out nationwide, except that within a decade the introduction of a compulsory national curriculum and standardised testing had begun to steer the education system back towards a one-size-fits-all approach. Within a few short years central government had essentially wrested the responsibility for education and its funding from local authorities and education had become a serious ‘political football’. Successive governments have focused on raising educational attainment as an indicator of their own effectiveness as a government and ironically that’s what’s resulted in SEN becoming a problem again in recent years.

Essentially, if you want an efficient one-size-fits-all education system and world-beating exam results it makes perfect sense to remove from the equation children who don’t fit into the system and are unlikely to do well in exams however hard everyone tries. That’s what the government did in the 1890s. If you want an education system that provides all children with an education suitable to their individual needs, you can forget about one-size-fits-all and world-beating exam results; you’ll need a lot of flexibility. That’s what the education system had developed into by the time of the Warnock committee. If you want both you’re likely to end up where we are now.

"Relativity" by MC Escher

“Relativity” by MC Escher

The Warnock committee defined special educational needs in terms of the educational provision ‘generally available in normal schools’. By definition, the better the provision in normal schools, the smaller the number of children who would be deemed to have special educational needs. The committee couldn’t have emphasised the need for SEN training for all teachers more strongly if it had tried, but perversely, the education system appears to have taken a step in the opposite direction.

teacher training

The Warnock committee recommended the inclusion of SEN training in the initial teacher training (ITT) for all teachers. Following the 1981 Education Act, the assumption that many children with SEN would be taught in mainstream schools and that all teachers would be trained in SEN led to the cessation of many special needs teacher training courses. They obviously haven’t been replaced with comparable training in ITT. This, coupled with the retirement of special education teachers and a reduction of the number of children in special schools, has meant that the education system as a whole has suffered a considerable loss of SEN expertise.

Reviews of SEN provision have repeatedly reported concerns about there being insufficient emphasis on SEN in ITT. But it’s only been since 2009 that Special Educational Needs Co-ordinators (SENCOs) have been required to be trained teachers, and only new SENCOs have been required to have SEN training. The current government has allocated additional funding for SEN qualifications (para 53) but only up until last year. This isn’t going to touch the problem. DfE figures for 2011 show that only around 7% of the total education workforce has SEN experience and/or training, and most of those people are concentrated in special schools. And special schools report ongoing difficulties recruiting suitably trained staff. This, despite the fact that the Warnock report 35 years ago pointed out that based on historical data, around 20% of the school population could be expected to need additional educational provision at some time during their school career. The report made it clear that all teachers are teachers of children with special educational needs.

Teachers’ expertise, or lack of it, will have a big impact on the attainment of children with SEN, but that hasn’t prevented government from developing unrealistic targets for all children under the guise of raising aspirations.

expectations of attainment

I mentioned earlier that over the last three decades education has become a ‘political football’. Concern is often expressed over the proportion of young people who leave school functionally illiterate or innumerate or without qualifications, despite evidence that this proportion has remained pretty constant for many years. In the case of literacy, it’s remained stubbornly at around 17%, by bizarre coincidence not far from the equally stubborn 20% figure for children with SEN.

But the possibility that some of those young people might be in the position they’re in because of lack of expertise in the education system – or even because they are never going to meet government’s arbitrary attainment targets and that that might actually be OK – doesn’t seem to have occurred to successive governments. In her keynote address to the inaugural national conference of the Autism Education Trust in 2009 the then Minister for Schools and Learning Sarah McCarthy-Fry, saw no reason why young people with autism shouldn’t achieve 5 A-C grade GCSEs. Some of course might do just that. For others such an aspiration bears no relation to their ability or aptitude, part of the definition for the ‘suitable education’ each child is required, by law, to receive.

Currently, funding for post-16 education requires young people to have or be studying for A-C grade GCSEs in both English and Maths. Post-16 providers are rolling their eyes. Although I can understand the reasoning behind this requirement, it’s an arbitrary target bearing no relation to the legal definition of a suitable education.

it’s the system

Currently, local authorities, schools and teachers are under pressure from the SEN system to make personalised, specialised educational provision for a small group of children, whilst at the same time the education system as a whole is pushing them in the opposite direction, towards a one-size-fits-all approach. This is a daft way to design a system and no matter how much effort individual professionals put in, it can’t work. But it isn’t the SEN system itself that needs changing, it’s teacher expertise and government expectations.

Over recent decades, successive governments have approached education legislation (and legislation in general, for that matter) not by careful consideration of the historical data and ensuring that the whole system is designed to produce the desired outcomes, but essentially by edict. A bit of the education system is wrong, so government has decreed that it should be put right, regardless of what’s causing the problem or the impact of changing part of the system without considering the likely consequences elsewhere.

In systems theory terms, this is known as sub-system optimization at the expense of systems optimization. That mouthful basically means that because all the parts of a system are connected, if you tweak one bit of it another bit will change, but not necessarily in a good way. Policy-makers refer to the not-in-a-good-way changes as unintended and unwanted outcomes.

The new SEN legislation is a classic case of an attempt at sub-system optimization that’s doomed to fail. It requires the education, health and social care sectors to do some joined up thinking and extend the support offered to children with SEND for a further decade – until they are 25 – at a time when all three sectors are undergoing massive organisational change and simultaneously having their budgets cut. It introduces personal budgets at a time when all three sectors are changing their commissioning arrangements. It fails to address the lack of expertise in all three systems. (Recent reports have pointed out that teachers aren’t trained in SEN, GPs don’t have paediatric training and children’s social workers don’t know about child development.) It fails to address the fundamental systems design problems inherent in all three sectors; a one-size-fits-all education system, and health and social care sectors that focus on cure rather than prevention.

This approach to systems design isn’t just daft, it’s incompetent and reprehensively irresponsible. People who have made hopeful noises about the new SEN system have tended to focus on the good intentions behind the legislation. I have no doubt about the good intentions or the integrity of the ministers responsible – Sarah Teather and Edward Timpson – but they have been swimming against a strong tide. Getting through the next few years will be tough. Fortunately, in the world of SEN there’s a lot of Dunkirk spirit – we’re going to need it.

References
Hegarty, S & Pocklington, K (1981). A junior school resource area for the visually impaired. In Swann, W (ed.) The Practice of Special Education. Open University Press/Basil Blackwell.
Warnock, H M (1978). Report of the Committee of Enquiry into the Education of Handicapped Children and Young People. HMSO.

progressively worse

‘Let the data speak for themselves’ is a principle applied by researchers in a wide range of knowledge domains, from particle physics through molecular biology to sociology and economics. The converse would be ‘make the data say what you want them to say’, a human tendency that different knowledge domains have developed various ways of counteracting, such as experimental design, statistical analysis, peer review and being explicit about one’s own epistemological framework.

Cognitive science has explored several of the ways in which our evaluation of data can be flawed; Kahneman, Slovic & Tversky (1982) for example, examine in detail some of the errors and biases inherent in human reasoning. Findings from cognitive science have been embraced with enthusiasm by the new traditionalists, but they appear to have applied the findings only to teaching and learning, not to the thinking of the people who design education systems or pedagogical methods – or those who write books about those things. In Progressively Worse Robert Peal succumbs to some of those errors and biases – notably the oversimplification of complex phenomena, confirmation bias and attribution errors – and as a consequence he draws conclusions that are open to question.

The ‘furious debate’

Peal opens Progressively Worse with a question he says has been the subject of half a century of ‘furious debate’; ‘how should children learn?’ He exemplifies the debate as a series of dichotomies – an authoritative teacher vs independent learning, knowledge vs skills etc. representing differences between traditional and progressive educational approaches. He then provides an historical overview of changes to the British (or, more accurately English – they do things differently in Scotland) education system between 1960 and 2010, notes their impact on pedagogy and concludes that it’s only freedom to innovate that will rescue the country from the ‘damaging doctrine’ of progressive education to which the educational establishment is firmly wedded. (p.1)

Progressive or traditional

For Peal, progressive education has four core themes;

• education should be child-centred
• knowledge is not central to education
• strict discipline and moral education are oppressive and
• socio-economic background dictates success (pp.5-7).

He’s not explicit about the core themes of traditional education, but the features he mentions include;

• learning from the wisdom of an authoritative teacher
• an academic curriculum
• a structure of rewards and examinations
• sanctions for misbehaving and not working (p.1).

He also gives favourable mention to;

subject divisions
the house system
smart blazers, badges and ties
lots of sport
academic streaming
prize-giving
prefects
pupil duties
short hair
silent study
homework
testing
times tables
grammar, spelling and punctuation
school song, colours and motto
whole-class teaching, explanation and questioning
the difference between right and wrong, good and evil
class rankings

I claimed that Peal’s analysis of the English education system is subject to three principle cognitive errors or biases. Here are some examples:

Oversimplification

For the new traditionalists, cognitive load theory – derived from the fact that working memory has limited capacity – has important implications for pedagogy. But people don’t seek to minimise cognitive load only when learning new concepts in school. We also do it when handling complex ideas. On a day-to-day level, oversimplification can be advantageous because it enables rapid, flexible thinking; when devising public policy it can be catastrophic because the detail of policy is often as important as the overarching principle.

Education is a relatively simple idea in principle, but in practice it’s fiendishly complex, involving political and philosophical frameworks, socio-economic factors, systems pressures, teacher recruitment, training and practice and children’s health and development. Categorising education as ‘progressive’ or ‘traditional’ doesn’t make it any simpler. Each of Peal’s four core themes of progressive education is complex and could be decomposed into many elements. In classrooms, the elements that make up progressive education are frequently interspersed with elements of traditional education, so although I agree with him that some elements of progressive education taken to extreme have had a damaging influence, it’s by no means clear that they have been the only causes of damage, nor that other elements of progressive education have not been beneficial.

Peal backs up with numbers his claim that the British education system is experiencing ‘enduring educational failure’ (p. 4). He says the ‘bare figures are hard to ignore’. Indeed they are; what he doesn’t seem to realise is that ‘bare figures’ are also sometimes ambiguous. For example, the UK coming a third of the way down the PISA rankings is not an indication of educational ‘failure’ – unless your definition of success is a pretty narrow one. And the fact that in all countries except the UK literacy and numeracy levels of 16-24 year-olds are better than those of 55-65 year-olds might be telling us more about the resilience of the UK education system in the post-war period than about current literacy standards in other countries. ‘Bare figures’ rarely tell the whole story.

Confirmation bias

Another concept from cognitive science important to the new traditionalists is the schema – the way related information is organised in long-term memory. Schemata are seen as useful because they aid recall. But our own schemata aren’t always an accurate representation of the real world. Peal overlooks the role schemata play in confirmation bias; we tend to construe evidence that confirms the structure of one of our own existing schemata as having higher validity than evidence that contradicts it, even if the evidence overall shows that our schema is inaccurate.

Research usually begins with a carefully worded research question; the question has to be one that can have an answer, and the way the question is framed will determine what data are gathered and how they are analysed to provide an answer. The data don’t always confirm researchers’ expectations; what the data say is sometimes surprising and occasionally counterintuitive. Peal opens with the question; ‘how should children learn?’ but it’s not a question that could be answered using data as it’s framed in terms of an imperative. That’s not an issue for Peal, because he doesn’t use his data to answer the question, but starts with his answer and marshals the data to support it. He’s entitled to do this of course. Whether it’s an appropriate way to tackle an important area of public policy is another matter. The big pitfall in using this approach is that it’s all too easy to overlook data that doesn’t confirm one’s thesis, and Peal overlooks data relating to the effectiveness of traditional educational methods.

Peal’s focus on the history of progressive education during the last 50 years means he doesn’t cover the history of traditional education in the preceding centuries. If Peal’s account of British education is the only one you’ve read, you could be forgiven for thinking that traditional education was getting along just fine until the pesky progressives arrived with their political ideology that happened to gain traction because of the counter-cultural zeitgeist in the 1960s and 1970s. But other accounts paint a different picture.

Traditional education has had plenty of opportunities to demonstrate its effectiveness; Prussia had introduced a centralised, compulsory education system by the late 18th century – one that was widely emulated. But traditional methods weren’t without their critics. It wasn’t uncommon for a school to consist of one class with one teacher in charge. Children (sometimes hundreds) were seated in order of age on benches (‘forms’) and learned by rote not just multiplication tables and the alphabet, but entire lessons, which they then recited to older children or ‘monitors’ (Cubbereley, 1920). This was an approach derived from the catechetical method used for centuries by religious groups and was understandable if funding was tight and pupils didn’t have access to books. But a common complaint about rote learning was that children might memorise the lessons but they often didn’t understand them.

Another problem was the children with learning difficulties and disabilities enrolled in schools when education became compulsory. The Warnock committee reports teachers being surprised by the numbers. In England, such children were often hived off into special schools where those deemed ‘educable’ were trained for work. In France, by contrast, Braille, Itard and Seguin developed ways of supporting the learning of children with sensory impairments and Binet was commissioned to develop an assessment for learning difficulties that eventually transformed into the Stanford-Binet Intelligence Scale.

Corporal punishment for misdemeanours or failure to learn ‘lessons’ wasn’t uncommon either, especially after payment by results was introduced through ‘Lowe’s code’ in 1862. In The Lost Elementary Schools of Victorian England Philip Gardner draws attention to the reasons why ‘dame schools’- small schools in private houses – persisted up until WW2; these included meeting the needs of children terrified of corporal punishment and parents sceptical of the quality of teaching in state schools – often the result of their own experiences.

Not all schools were like this of course, and I don’t imagine for a moment that that’s what the new traditionalists would advocate. But it’s important to bear in mind that just as progressive methods taken to extremes can damage children’s educational prospects, traditional methods taken to extremes can do the same. It’s difficult to make an objective comparison of the outcomes of traditional and progressive education in the early days of the English state education system because comparable data aren’t available for the period prior to WW2, but it’s clear that the drawbacks of rote learning, whole class teaching and teacher authority made a significant contribution to progressive educational ideas being well-received by a generation of adults whose personal experience of school was often negative.

Attribution errors

Not only is the structure of some things complex, but their causes can be too. Confirmation bias can lead to some causes being considered but others being prematurely dismissed – in other words, to wrong causal attributions being made. One common attribution error is to assume that a positive correlation between two factors indicates that one causes another.

Peal attributes the origins of progressive education to Rousseau and the Romantic movement, presumably following ED Hirsch, a former professor of English literature whose specialism was the Romantic poets and who re-frames the nature/nurture debate as Romantic/Classical. Peal also claims that “progressive education seeks to apply political principles such as individual freedom and an aversion to authority to the realm of education” (p.4) supporting the new traditionalists’ view of progressive education as ideologically motivated. Although the pedagogical methods advocated by Pestalozzi, Froebel, Montessori and Dewey resemble Rousseau’s philosophy, a closer look at their ideas suggests his influence was limited. Pestalozzi became involved in developing Rousseau’s ideas when Rousseau’s books were banned in Switzerland. Pestalozzi was also influenced by Herbart, a philosopher intrigued by perception and consciousness, topics that preoccupied early psychologists such as William James, a significant influence on John Dewey. Froebel was a pupil of Pestalozzi interested in early learning who set up the original Kindergärten. Maria Montessori trained as a doctor. She applied the findings of Itard and Seguin who worked with deaf-mute children, to education in general. The founders of progressive education were influenced as much by psychology and medicine as by the Romantics.

Peal doesn’t appear to have considered the possibility of convergence – that people with very different worldviews, including Romantics, Marxists, social reformers, educators and those working with children with disabilities – might espouse similar educational approaches for very different reasons; or of divergence – that they might adopt some aspects of progressive education but not others.

Peal and traditional education

Peal’s model of the education system certainly fits his data, but that’s not surprising since he explicitly begins with a model and selects data to fit it. Although he implies that he would like to see a return to traditional approaches, he doesn’t say exactly what they would look like. Several of the characteristics of traditional education Peal refers to are the superficial trappings of long-established independent schools – bells, blazers and haircuts, for example. Although some of the other features he mentions might have educational impacts he doesn’t cite any evidence to show what they might be.

I suspect that Peal has fallen into the trap of assuming that because long-established independent schools have a good track record of providing a high quality academic education, it follows that if all schools emulated them in all respects, all students would get a good education. What this view overlooks is that independent schools are, and have always been, selective, even those set up specifically to provide an education for children from poor families. Providing a good academic education to an intellectually able, academically-inclined child from a family motivated enough to take on additional work to be able to afford the school uniform is a relatively straightforward task. Providing the same for a child with learning difficulties, interested only in football and motor mechanics whose dysfunctional family lives in poverty in a neighbourhood with a high crime rate is significantly more challenging, and might not be appropriate.

The way forward

The new traditionalists argue that the problems with the education system are the result of a ‘hands off’ approach by government and the educational establishment being allowed to get on with it. Peal depicts government, from Jim Callaghan’s administration onward, as struggling (and failing) to mitigate the worst excesses of progressive education propagated by the educational establishment. That’s a popular view, but not necessarily an accurate one and Peal’s data don’t support that conclusion. The data could equally well indicate that the more government intervenes in education, the worse things get. The post-war period has witnessed a long series of expensive disasters since government got more ‘hands on’ with education; the social divisiveness of the 11+, pressure on schools to adopt particular pedagogical approaches, enforced comprehensivisation, change to a three-tier system followed by a change back to a two-tier one, a constantly changing compulsory national curriculum, standardised testing focused on short-term rather than long-term outcomes, a local inspectorate replaced by a centralised one, accountability to local people replaced by accountability to central government, a constant stream of ‘initiatives’, constantly changing legislation and regulation and increasing micro-management.

A state education system has to be able to provide a suitable education for all children, a challenging task for teachers. The most effective approach found to date for occupations required to apply expertise to highly variable situations is the professional one. Although ‘professional’ is often used simply to denote good practice, it has a more specific meaning for occupations – professionals are practitioners who have acquired high-level expertise to the point where they are authorised to practice without supervision. Regulation and accountability comes via professional bodies and independent adjudicators. This model, used in occupations ranging from doctors, lawyers and architects to builders and landscape gardeners, although not foolproof, has worked well for centuries.

Teaching is an obvious candidate for professional status, but teachers in England have never been treated as true professionals. Initial teacher training has often been shortened or set aside entirely in times of economic downturn or shortages of teachers in specific subject areas, and it’s debatable whether a PGCE provides a sufficient grounding for subject-specialist secondary teachers, never mind for the range of skills required in primary education. Increasing micromanagement by local authorities and more recently by central government has undermined the professional status of teachers further.

I see no evidence to suggest that the university lecturers and researchers, civil servants, local authorities, school inspectors, teaching unions, educational psychologists and teachers themselves that make up the so-called ‘educational establishment’ are any less able than government to design a workable and effective education system – indeed by Peal’s own reckoning, during the period when they actually did that the education system functioned much better.

Despite providing some useful information about recent educational policy, Peal’s strategy of starting with a belief and using evidence to support it is unhelpful and possibly counterproductive because it overlooks alternative explanations for why there might be problems with the English education system. This isn’t the kind of evidence-based approach to policy that government needs to use. Let the data speak for themselves.

References
Cubberley, EP (1920). The History of Education. Cambridge, MA: Riverside Press
Gardner, P (1984). The Lost Elementary Schools of Victorian England: The People’s Education. Routledge.
Kahneman, D., Slovic, P & Tversky A (1982). Judgement under Uncertainty: Heuristics and Biases. Cambridge University Press.
Peal, R (2014). Progressively Worse: The Burden of Bad Ideas in British Schools. Civitas.

public services and systems thinking

Last week I attended a conference hosted by Stoke-on-Trent City Council called A radical approach to reshaping public services. It was one of the most informative I’ve ever been to, so I thought a summary might be useful to anyone struggling to navigate public sector services.

A team from the city council and another from Bromsgrove and Redditch councils explained how they are applying Vanguard’s systems thinking approach to the way they support local people. The two teams have tackled local issues slightly differently; in this post I’ve amalgamated what they described, to give an overview. Obviously, this is my own overview and I’m happy to be corrected if I’ve got anything wrong.

Although both teams use the Vanguard approach, systems thinking for organisations wasn’t invented by Vanguard, but is based on the principles of systems theory. Systems theory is pretty robust. We know how systems work in many different domains. Because organisations are systems, applying systems theory to them makes a lot of sense. (Management theories, by contrast, usually address only part of the organizational system, and that’s why they tend not to work very well.)

Some key principles of systems thinking as applied to organisations

Form follows function

If you want your organization to be effective, you need to have a clear idea of its function. If its ultimate goal is to make sure local people can get on with their lives (the primary purpose of local authorities) you need to have valid, reliable and relevant information about what services people need to enable them to do that. Then you can decide what things your organization has to do to meet those needs (function) and what structure will best enable it to do those things (form).

Systems should be designed as whole systems

If you’re designing a system, you need to make sure the whole system is actually doing what it’s supposed to do. A system is essentially a set of interacting components. If you change one component, there’s a good chance another component will be affected, so you need to check out the impact on the system as a whole, or the system as a whole might not work.

Good data about the nature of demand is essential

“Demand’ is anything that takes energy out of the system. It includes the processes the system carries out, the number of people using it and the complexity of their needs, the cost of staffing, buildings and equipment etc. It also includes what Vanguard calls ‘failure demand’ – demand due solely to the failure of the system to function efficiently. Failure demand would include duplication, re-referrals, complaints etc.

Variation requires adaptability

A system dealing with high variation (e.g. a large population with a wide range of needs) must be adaptable if it’s to respond appropriately to that variation. The best way we’ve found so far of meeting the wide range of needs across a community is through the classic model of professional expertise. For millennia, communities have met needs through access to people with high levels of expertise, whether carpenters or doctors, tailors or teachers. The key features of these professionals are that they have a high level of both specialist expertise and autonomy. Both qualities are characteristic of the teams using the systems thinking approach.

How do you apply systems thinking to a public service organisation?

Stoke-on-Trent City Council was faced with escalating problems – unemployment, rent arrears, anti-social behaviour etc – at the same time as experiencing significant reductions in funding. It was clear that even before funding cuts, whatever the council was doing to tackle the problems wasn’t an unqualified success, so a radical change was needed to avert disaster. The council applied the systems thinking approach in three phases;

• an analysis of the effectiveness of the old system,
• a pilot study of the systems thinking approach in one location, and
• scaling-up the new system informed by data gathered from the pilot study.

Phase 1: Analysing the effectiveness of the old system

Because it was clear that the old system a) wasn’t working efficiently and b) wasn’t financially sustainable, a detailed examination of its strengths and weaknesses might, on the face of it, appear to be a waste of time. It turns out that several useful outcomes emerge from a detailed analysis of current practice. A analysis enables you to;

• look at the system as a system, not as a set of disconnected agencies and departments,
• see how effective the system is, and why it is or isn’t effective, and
• get detailed information about costs.

The city council already had good city-wide data, so they knew where most of the problems were located geographically, but they didn’t know much about the people with the problems. They couldn’t answer the question “Where do the people who go through your system end up?”

The council team looked in detail at a small number of complex cases from the point of view of the people involved. Analyses carried out by a multi-agency team shed light on how the old system worked – or rather how it didn’t work. The analyses included mapping the pathways followed by people using the system, creating a timeline of interactions with agencies, and comparing what people wanted with what they got.

Mapping the pathways followed by people using the system

Typically, when someone first engaged with the system, the department they contacted dealt with issues within their remit, and then referred the person to other departments or agencies for any other issues. To analyse the pathways people followed, the team drew a map, for each case study, showing each referral. They ended up with some hugely complex diagrams, showing that people often ended up in loops of referrals and re-referrals to the same agencies. Anyone unfamiliar with the system would have found it bewildering and frustrating. Confusion and frustration is what people using public sector systems often report, but until you’ve seen it mapped out on paper, it’s difficult to see where the specific causes of the confusion and frustration lie.

Diagram showing how referrals were mapped

Diagram showing how referrals were mapped

Mapping out a time-line of interactions with different agencies

The team also looked at what interactions people had with different agencies and when. They used used post-it notes to represent interactions, colour-coded for the relevant agency, on a timeline. A typical pattern consisted of a few interactions with a single agency for a few months that then rapidly fanned out into many interactions with multiple agencies. What was clear was that there was no ownership of the case by any specific agency.

Diagram showing timeline of interactions

Diagram showing timeline of interactions

Asking people what they wanted

The team compared what support people said they wanted with what they actually got. One woman, who ended up with health and mobility problems, no support and her children in care, initially had two ‘wants’; access to the first floor of her home and help with housework. If those ‘wants’ had been met at the outset, the cost to the council would have been relatively low. As it was, the fact that neither ‘want’ was addressed resulted in an outcome that was very expensive for the council and catastrophic for the family.

What these exercises showed is that;

• the problems that people presented with (e.g. rent arrears) were often outcomes of other problems
• people often accurately identified the root causes of problems given the opportunity to do so
• addressing the root causes promptly could result in significant financial savings and avoid problems becoming complex
• people using services often experienced unnecessary referrals to multiple agencies and many agencies were duplicating each other’s work
• each interaction with services added to costs
• each increase in complexity of problems added to costs.

Phase 2: Pilot study in one locality

An area of the city known to have high needs was chosen for a pilot study. A ‘locality’ team was set up to respond to issues raised by people in this area. The team’s brief was to address any problems identified when people living in this area came into contact with any local authority services or with the fire or police services, which were by now actively involved with the project.

The new locality team differed from the old functional teams in several important respects;

• each team member would be responsible for ensuring the support of a small group of citizens
• they ‘pulled in’ additional expertise to the team if required, rather than referring the citizen out to another agency
• they used measures to monitor performance and outcomes but didn’t use targets

Phase 3: Evaluating the effectiveness of the pilot study

After six months, the pilot study was evaluated at three levels, Tier 3 – the city level, Tier 2 – which could be described as the locality level, and Tier 1 – the individual level.

Tier 3: City level evaluation

Two years prior to the pilot study the council had identified 35 key strategic measures in order to monitor the cost-effectiveness of their services, so they already had good data at the macro-level, Tier 3. The team used the ‘triangle of need’ to identify the level of support required by particular households.

triangle of need

Triangle of need

Small, but clear reductions in costs for the pilot area of the city were evident compared to similar localities used as controls. The results are tentative because the intervention had only been in place for 6 months.

Tier 1: Individual level evaluation

During the pilot study, the outcomes for seven people were assessed in detail, so data were then available at the individual level – Tier 1. The results were not what the team expected. Overall, the number of interactions with local authority services had increased, but the cumulative cost of those interactions had gone down slightly. By contrast, the demand on health and police services decreased significantly – in some cases by around 90%. Fire service interactions increased, but costs also plummeted because fire service involvement was around fire prevention, not emergency call-outs.

The seven citizens were asked at the beginning of the pilot study what problems they wanted to resolve and their perception of their progress towards resolution was mapped on radar (spider) charts. After six months most had seen significant improvements. The locality teams also filmed some quite moving interviews with people who’d received support from them. The citizens in question were obviously grateful to the locality teams for providing them with relevant, timely support. And surprised that the council offered that support at all.

Tier 2 – Locality level evaluation

After the first 6 months, the proportion of households in the pilot locality needing multiple-agency or specialist support had dropped from 35% to 20%. The council estimates that there are around 5,000 households in the city that fall into one of these two categories and that if the improvements seen in the pilot study can be scaled-up city-wide, over 5 years the council could save £40-80m.

Comments

Some points worth noting emerged during the course of the conference.

Getting other agencies on board was essential

Stoke city council were clear that they couldn’t have proceeded with this work without the active involvement of the police and fire services, and that getting the support of people in charge of other agencies was essential. The massive cost savings to the police, fire and health services suggested their involvement was a worthwhile investment.

Local councillors, often initially sceptical about yet another re-organisation, generally found the facts and figures persuasive.

Seeing the situation on the ground is essential

One housing officer was shocked when she visited the properties that tenants were expected to occupy. So were other team members when they found out what had happened to people who’d contacted their departments. What’s on paper doesn’t always match real life.

It takes time to build up the trust of people using the new system

Finding out more about people’s problems can seem intrusive, and it took time to build up the trust of those with complex problems. But word-of-mouth recommendations about timely, effective support and positive outcomes is beginning to change the relationship between local people and the council.

Staff using the systems thinking approach wouldn’t go back to the old system

Council officers found their working life transformed by the increase in variety, autonomy and effectiveness of their new roles. Most wouldn’t go back to the old way of working. One important factor in improving the time taken to respond to problems, was that the members of the locality team (including a fire officer, a police officer and an alcohol support worker) were based in the same room, resulting in almost instant communication. Other agencies had a designated member of staff to deal with locality team queries.


The main problem with systems thinking – it looks like common sense

One of the drawbacks of systems thinking is that it looks so obvious, it’s easy for organisations to think they are using it already .

For example, many local authority children’s services use multi-agency teams and emphasise the importance of ‘joined-up thinking’, but the teams, their thinking and the outcomes that result bear little resemblance to the locality teams or the outcomes they’ve achieved.

I’ve also seen the triangle of need used by another local authority, not as a way of representing data, but to categorise children with disabilities (low, medium and high needs). Support is available only to families with children with severe or complex needs – those in the top section – meaning there’s a real risk that the problems of families in the middle section will be allowed to escalate.

Systems thinking is based on tried-and-tested principles. It can cope with highly variable demand and results in increased job satisfaction, reduced costs and improved quality of service. The savings and improvements can be significant. A real ray of hope in a public sector facing its greatest ever challenges.