Summary of my EMA posts

Educational Maintenance Allowance is an allowance of between £10 and £30 a week paid to 16-19 year olds from low-earning backgrounds who stayed on in further education. Its original aims were to increase participation and achievement in further education. It cost about £550m a year.

The coalition government have scrapped it and replaced it with a much smaller pot of money to be paid to students in hardship. The NUS, Polly Toynbee, a range of Labour MPs and the website Save EMA have campaigned for it to be saved.

My experience as a teacher is that it is an extremely ineffective policy.  When I read the relevant research, it all completely confirmed my concerns. Here is a summary of the posts I have written supporting the EMA cut. 

1. EMA has extraordinarily high ‘deadweight’ costs – that is, most of the people receiving the benefit would have stayed on in further education even without it. The NUS have consistently misrepresented this despite a phenomenal weight of evidence proving it.
2. Even those who have been persuaded to stay on because of EMA have not achieved very well as a result. Nor have those receiving who would have stayed on anyway benefitted by being able to do less paid work and concentrate on their schoolwork. The evidence on achievement is weak, and the most positive thing that can be said is it improves students’ grades by about one eighth of one A-level grade. Anthony Painter has questioned the stats in the above two posts, and my response to him can be found here, here and here.
3. One of the main things EMA was meant to pay for was transport costs – yet data show that 50% of further ed students live within half hour’s walk of a post-16 establishment. Many areas also have free or subsidised transport for 16-19 year olds.
4. The above facts have been consistently misrepresented in the media, mainly by the NUS and the Save EMA campaign, but also by Bridget Phillopson, Lisa Nandy, Polly Toynbee (one, two, three), Ken Livingstone and Hazel Blears.This has meant that anyone who questions the value of EMA is represented as wanting to destroy opportunity, often in quite lurid and sensationalist terms.
5. In particular, the NUS misrepresent the issue by confusing percentages and percentage points. The head of the Save EMA campaign also shows a poor grasp of those statistics. 
6. There is anecdotal evidence of EMA being spent on non-essential and non-educational items. Research evidence points towards transport being what students spend most of their EMA money on – but given the fact about proximity to college, and given the significant anecdotal evidence, I suggest that a lot of this expenditure may be on driving lessons and cars, which certainly counts as transport but is hardly an essential.
7. There is plenty of anecdotal evidence that suggests EMA negatively affects students’ attitude to learning, and indeed much of the defence of EMA sees students as entirely passive creatures who lack all personal agency. This was shown again by the students themselves at the recent protests.
8. The replacement scheme will offer many of the benefits whilst cutting a lot of the deadweight.
9. I think that EMA is probably the quintessential new Labour social programme. 
10. Finally, in case anyone thinks I really am being a callous git who doesn't understand how hard it is to be poor, here is my own experience of EMA, which I was in fact eligible for.
11. The IFS seem to be about to bring a new paper out about it - I haven't had a chance to read it but here are my thoughts based on some of the advance info out there.

The IFS on EMA, again, courtesy of Channel Four

I am writing this early Tuesday morning. There is an interesting article on the Channel Four blog which suggests that later today the IFS are going to publish a paper suggesting that the costs of EMA are more than recouped.  Channel Four’s blog is the only one to feature this, and from the article it seems as if the details of the paper have been leaked by the report’s author, Haroon Chowdry, to the Channel Four blogger, Faisal Islam.

I’ll deal with what we learn about this report in a minute, but first of all let’s take Islam’s comments on it.  I’d be interested to know how many of his fairly partisan comments are shared by the author of the paper who seems to have leaked it, and how many are his own. Anyway, that aside, he takes an extremely negative tone towards the government that in some cases are absolutely not justified by the facts and in others, well, we’ll have to wait and see what the report actually says.

So, Islam’s comments. He says that

 ‘the Institute for Fiscal Studies will tomorrow fire a flaming bolt through the middle of some rather second rate analysis put out by the Government in favour of scrapping this policy.’ 

Which second rate analysis would this be? The analysis by Policy Exchange of the effectiveness of EMA, which cited analysis by...the IFS? The Policy Exchange report was written by Sam Freedman, who is now Michael Gove’s policy adviser, so it seems fair to say the government are very well aware of the first-rate analysis done by the IFS on this issue - which backs up their ideas.

Then Islam says this:

That is indeed the finding of the only peer-reviewed analysis of the EMA, done by the IFS team, but sadly unavailable online as it is an article for the Journal of Human Resources (hat tip to the FT’s Christopher Cook). Tomorrow’s IFS analysis is likely to reaffirm that finding.

Hmm. I automatically assumed that all the IFS’s reports were peer-reviewed, in which case there are a whole library of them here, freely available, on their website. Which 'only peer-reviewed analysis' is Islam talking about? Perhaps I’m wrong and these ones on the IFS website are not peer-reviewed.  The analysis I have leant on when doing my posts on this blog has been this one here, which as far as I know is the most recent and the only one to take attainment into account, not just participation.

Unfortunately, I am going to be busy at work for most of the rest of this week, so I am not going to be able to read this report which is published tomorrow. I’ll just flag up a few issues I have with it based on what Islam quotes Chowdry as saying. I am not seeking to question any of the IFS’s statistics – as far as I know anything about them their intellectual credibility is beyond doubt, which is why I used their stats when making my blog posts on this.  But as my discussions with Anthony Painter have shown, stats are only one aspect of a case. Good people should be able to agree on stats, and give or take a few salient points I think Painter and I largely do. Once the stats have been established, that’s actually when the interesting stuff starts. Anthony Painter thinks that a deadweight cost of 90% is fine, I don’t.  Our differences are, to use a word much in vogue of late, “ideological”. That is, we’re both ideological, not, as Labour would want you to believe, that they are the staunch upholders of evidence-based rational politics and everyone else wants to change things according to silly ideas of morals and things.  None of this is to argue for a second that the stats don’t matter.  Of course they will affect your understanding of an issue. That was why I was so keen to correct the NUS on their ridiculous error here. Many more people will think that EMA is worth it if they think the NUS’s stats are right than if they think Anthony Painter’s stats are right. If you say that 65% of kids will drop out if EMA is pulled (because, unbelievably, despite claiming to represent students, you can't tell the difference between a percentage and percentage point), then many more people will support it than if you say that 10% will. The stats do matter, but even when they have been established, that will not solve every disagreement, and nor should it.

So, the IFS are important in this argument because they can provide unimpeachable stats. But as I say, those stats can and will be interpreted in different ways, as I will show below. As well as this, I, and my fellow teachers, are important in this argument, because we can provide something the IFS cannot – evidence of the moral impact of EMA, which I deal with not in this post but here and here. Now, because of these moral qualms I have about it, then for me what counts as statistical 'success' of the EMA will probably be set a lot higher than for someone simply looking at the figures. Like I say, call me ideological.

To give an extreme example of what I mean by this, let's assume that I run a rigorous and controlled experiment that proves conclusively that shooting the kid who comes bottom in a test results in greater attainment in that test for the entire class. My facts and stats are unimpeachable, but it doesn't matter: no right thinking person would agree that this was a good policy. Its one 'cost' is greater than any possible benefit to attainment, however impressive. Now of course EMA isn't like shooting kids - like I say, this is an example. But I believe that EMA has many very significant drawbacks which aren't captured in the stats, so whilst I would be prepared to support it, for me the attainment/participation/future earnings/future wellbeing stats would have to be really, really good for me to do so. Plus, any programme, however successful, also has to justify its efficiency - could you achieve the same results for less? So even if we had a programme that cost, say, about 100,000 but instantly generated benefits worth ten times that, you could still argue that the programme should be reformed if you felt that only 90% of the money was getting you the benefit and you'd found a way to cut it that didn't affect the overall output.

OK, onto those facts:

“The initial outlay of the EMA policy is likely to be more than recouped by the increase in productivity that we expect to result from the 16- and 17-year-olds staying on in education for longer”. 

“is likely?” This is the first hint you get here that the actual findings of this report are not nearly as confident or secure as that 'flaming arrow' quote by Islam would have you believe. So the first thing I would want to do when I see the paper is to see how they’ve decided this.  I’d also be interested to see what the causal links are between 16 and 17 year olds staying on in school in longer, and an increase in their productivity.  This seems to me a bit like the reasoning that went around a few decades ago about degrees. People getting degrees earn more money, the reasoning went, so let’s encourage more people to do degrees and they will earn more money. More people now do degrees, and the graduate wage premium has fallen. Just because there may have been an increase in productivity associated with those who stayed on at school longer, does not mean that this will hold true if we increase the numbers who stay at school.

But it is a strong piece of evidence, that even if relatively small proportion stay on at school as a direct result of the EMA (and there are arguments about these numbers), it could still be a net benefit to the economy in the long run.

Note that 'could' again - not sure how reliable any of these statements are, again we have to wait for the report. This point seems to be accepting that EMA does only benefit a small proportion of those who get it. I find it particularly hard to believe in this case that the benefit to this small proportion is enough to eventually pay back the entire costs of EMA to everyone. The most generous IFS calculation I’ve seen of the deadweight cost is about 93%. Other less rigorous studies show 83%. Let’s be really generous and say it’s 80%. The increased earnings of 20% of kids are going to pay back the EMA costs of the entire 100%? That seems really implausible.  The cost of EMA is 550m a year, how long would it take for the net extra earnings of the 20% to reach that sum?  600k are on EMA, 20% of that is 120k, so if you could prove those 120k kids left college and instantly earnt 5k more than they would have done without, that would pay back the entire year’s costs of EMA and all their future increased earnings would be ‘profit’. But of course, it would only pay it back to the individuals, not to the Exchequer, as this post makes clear. So the taxpayer would still be out of pocket. Take a tax rate on that extra 5K of 33%, and it would take three years (five when you include the two where you were doling the cash out) before the scheme broke even, and then you would have to control for inflation over the five years since you started doling it out, so let's call it six years and that is before you start to think about kids going to uni - which might increase the wage premium but also delays it. All the tax on the kids’ extra earnings after that would be ‘profit’, or return on investment. Let’s assume that if the government invested the 550m in gold or bonds or used it to pay down debt or used it on another social programme – let’s say the govt could easily get 2% on that sum after inflation. In the six years it took EMA to break even you’d be well up with your alternative investment - compounded, you'd be 120m up by the time EMA broke even. And that’s on a wildly optimistic scheme that sees more kids benefitting from EMA than any study has shown and benefitting by a sum of money which I think is highly improbable – I don’t know any 18 year olds who earn 5k more than 16 yr olds by dint of the two years they did at college.  Now I don’t actually know that is what the report is saying, because of course I don’t have it. Perhaps it is saying that the increased earnings and productivity of 20% of kids will pay back the EMA costs of that 20%, which is an entirely different and far less impressive claim as it would still leave EMA overall costing an awful lot more than its benefits, and would push that notional break-even point I speak of above even further away, even on the most optimistic scheme.

Anyway, my essential point here is that EMA may very well give a long term return in its investment, in the same way that putting your money in a bank account paying 0.5 interest will give you a return.  It’s the scale of the return that matters. EMA may very well break even or turn a profit, but it has very significant opportunity costs – you could get a much better return on investment elsewhere.

There is of course a 'moral' or 'ideological' rejoinder to this that I am sympathetic to. Even if you could make 120m more elsewhere, isn't it worth 120m to give people a better job and better lifestyle? I am definitely sympathetic to this idea, which is why if I were reinvesting this money, I would not use it to pay down debt or to buy gold - I would use it in education, in a programme that could give you a much better short and long term return on investment - probably one involving primary or secondary, probably akin to...ooh...let me see...the Pupil Premium! And of course the other thing this report doesn't seem to take into account is that EMA isn't really being completely abolished - there's going to an extra 50 odd million in an emergency pot of money for needy kids. So any comparison would have to take into account the fact that at least some of this money will find its target - and probably a far greater percentage will find its target from this budget than from the EMA one.

One final thing:

As the IFS told me tonight: “This cost-benefit calculation does not include other benefits through other routes: for example, increases in educational attainment that might arise for the EMA recipients who would have stayed in education anyway, but now have more time to devote to studying.”

I can't believe the IFS have said this when their previous study specifically looked at the effect of EMA on the attainment of all kids receiving it. This is very baffling.As I have shown in a previous post, the increases in attainment achieved by kids on EMA are statistically significant, but not educationally significant - that is, I would be pretty confident of replicating them if you paid to send every exam paper off to be remarked. And of course any analysis of attainment is inevitably and unfortunately constrained by the fact that the English exam system is almost certainly not a reliable metric. I know this is an entirely separate argument, but it really does seem to me that the recent OECD figures go some way to proving the gut instinct a lot of people in education have that exams are getting easier.

Unfortunately I am going to be very busy for most of the rest of this week and probably won't have any time to read the paper until the weekend. I've written this quickly and whilst tired and it is all fairly tentative so there may unfortunately be some errors, but I hope it expresses the gist of my ideas.

The dartboard theory of public spending

Overall, the establishment and defence of EMA are examples of what I might call the dartboard theory of public spending, where the dartboard represents the social problem to be fixed and the blind dartsman represents the government
1. Spot dartboard
2. Chuck 1000 darts in general direction of dartboard.
3. Commission IFS to do cross-cohort longitudinal study showing that three darts did, indeed, hit the double top.
4. Claim you are therefore the Phil Taylor of solving social problems.
5. Demand an extra 1000 darts in order to get even better results.
6. Claim all those against this idea are against doing anything about social problems at all, ever – and even better claim any cut to the 1000 darts is evil, wicked, devastating, punitive, cruel, unfair, pernicious, a tragedy and a travesty.

I know who I'd rather have in charge of public spending.

Further response to Anthony Painter

I am in italics.

Thank you for the response. And you are right to say that my major issue is with your analysis rather than your statistics.

Yes, ultimately I agree that our disagreement is analytical, not statistical, but some of the things you say below, whilst not actually impacting that much on either, are still such misrepresentations that I have had to comment about them.

The problem I have with your statistical ‘dancing around’ is that in previous posts you have flipped around between pilot area averages and EMA recipient averages without making it clear what you were quoting and why. That weakens your case considerably.

Does it? Does it really? I assumed my readers had some intelligence – which you clearly do – and that weakens my case? If there was a discrepancy of several times in the stats I was quoting, then you would have a point, but in my follow up post I do an average of all the EMA recipient averages and it comes out at 5%, which is even less than my original post claimed. The reason why I jumped around a bit was actually because I was following the lead of a lot of EMA defenders, such as those in the NUS and CfBT, who seem to have dived into the paper and picked the biggest percentage they could find. Because I was dealing with their analyses, I obviously needed to use the figures they’d picked so I could follow through on their analyses. So when I jumped around, yes, actually, I was ‘weakening my case’ in a different sense to the one you mean because I was deliberately letting my opponents pick the figures that best backed up their argument. Show me one place where my jumping around resulted in a significant discrepancy that favoured me?

The problem I think you are encountering is that you are trying to two things at the same time and, as a consequence, you end up achieving neither. The first thing you are trying to do is administer a statistics lesson (to the NUS presumably.) But your posts have demonstrated how easy it is- through ill-discipline-

!!!

to give a distorted view.

Right, just because I jumped around DOES NOT mean I distorted. Again, show me one area where my jumping around significantly distorted the case? The only places I can find where it did are, as I state above, where I distort in favour of EMA in order to track the analyses of pro-EMA reports.

Your clear statistical capability is masked by unclear presentation which devalues

How does it devalue?

that element of your case. (and thanks for the patronising aside on Appendix D! I did as it happens read Appendix D though Table 1 is pretty self-explanatory anyway and the accompanying text that I quoted makes it equally clear. It should be noted that Table 1 (as opposed to 1a and 1b) deals with participation only and not attainment.)

OK fair enough, that may have sounded a little patronising. But if you had read appendix D, why did you say in your original post:

The difference between the area and individual impact of EMA.

The figures you mainly quote above are pilot area averages (thou you do jump about a bit). It is important to read pp 6-7 of the IFS report to understand why this is problematic. The key point is:

"We therefore suggest as a rule of thumb that, in order to obtain the effect of the EMA across those who received it, the estimates above be multiplied by a factor of 2½ (for outcomes at age 16) or 3 (for participation at age 17 and attainment outcomes)."

So the participation and attainment impacts are in the main much greater than you suggest.

That bit above in bold is the only thing you said in your original post about area/individual impacts, and you do say:

 So the participation and attainment impacts are in the main much greater than you suggest.

When in fact as I showed in my first response and as you would have known had you looked closely at table one and appendix d that therefore the participation impacts WERE NOT much greater than I suggested and DID NOT have to be factored up.

In particular, you criticise the NUS (rightly in fairness)

I am still waiting for their response. And this is not a trivial matter – as much as you think EMA matters even on the basis of the facts you cite, the truth is that there is a massive difference between what you as a numerate person are claiming about EMA and what the NUS are claiming. Now, you are still backing EMA despite the fact that you realise it is much less effective than the NUS claim, because you still think it’s effective enough. But I think if everyone were clear about how effective you think it is statistically, then there would be a lot fewer people willing to back it.

 on their confusion of percentages v percentage rates. But yet you flip between the two over the posts here without making it clear what you are doing and why. In your case, it’s not confusion; it’s lack of clarity.

Final point on the analysis, the point you make about the impact of EMA on different ethnic groups is, well, ill-considered. A greater proportion of certain groups receive it and so by looking at these sub-groups we can get a better sense of its real impact (though there may be issues specific to certain groups, it is highly likely their socio-economic status is the major impact on likelihood to participate in education.) So yes a majority of people who receive EMA are white British but a majority (probably) of, say, black African British receive it. My point was quite simply that if you want to understand the real impact of the policy you’d be better to look at the impacts on the latter than the former group. So it’s nothing to do with racial targeting etc. Quite why you went down that route is difficult to fathom.

Not sure I get what you mean here – as I’ve established, for participation I am only looking at EMA recipients, not the entire black or entire white communities. This analysis seems to show that it does have a bigger impact on ethnic minorities who receive it as opposed to white students who receive it.

Part 2:

Anyway, the major issue with the analysis is not actually to do with statistics- though it should be emphasised the issues presented by your use of statistics of are not insignificant. It is your reliance on the ‘deadweight’ argument. 

All public programmes have a degree of ‘deadweight.’

Yeah, you’re right. A degree! Not 90 percent! If that’s a ‘degree‘ I’d hate to see what you thought a ‘large degree’ was.

The example I gave on my piece on Left Foot Forward was on GP check ups. Only a tiny proportion lead to the identification of a serious illness so the vast majority is deadweight so would you cut this expenditure? 

This is a terrible comparison that tells us barely anything about EMA.  Firstly, just because only a tiny proportion of GP check ups lead to the identification of a serious illness, does not in any way mean the rest are wasted!  Frequent GP check ups have a preventative function- surely you realise this? Secondly, even if you are sceptical about how easy it will be to identify the kids who don’t need EMA (I deal with this point below) surely you’ve got to realise it’s a damn sight easier to work out kids who don’t need EMA than it is to work out a stomach pain that’s cancer or a stomach pain that’s a dodgy bit of meat? Even the CBfT paper which supports the EMA fully concedes that you could abolish the two lower EMA bands without significantly affecting the drop out attainment rate – can you at least accept that? That there are very easy and simple cuts you can make that won’t lead to anyone dying of cancer?

You could say the same about education- more than half don’t leave school with 5 GCSEs A-C (including mathematics and English) so why not just not bother with the rest and put them on some cheap literacy and numeracy programme with some basic subject matter, teach them two hours a day, and reduce costs by over 50%? There is nothing intrinsic in your statistical analysis to suggest there is too much ‘deadweight’ in the programme.

Another dreadful comparison. I happen to believe every kid is capable of getting 5 A*-Cs. Even the ones who don’t – I believe in my naivety that they get some value out of doing GCSEs anyway. The money we spend on them is not deadweight. The money we spend on GP check ups that don’t lead to identifications of cancer is not deadweight. I don’t think you quite get what the definition of deadweight is. Literally, ‘Expenditure to promote a desired activity that would in fact have occurred without the expenditure.’

I’ll give you some other examples of programmes that have deadweight in, and I think there is a good case for cutting a lot of them too. Winter fuel for pensioners is probably the best example. Here, the desired benefit is that all pensioners should be warm in winter, when in fact lots of them would without the cash. Likewise a lot of the pensioner benefits fall into this category – TV licences, travel. Child benefit too – although here it is less clear because there are differing claims about what the actual desired outcome of child benefit is. If it was to prevent child poverty or give all kids a little treat now and again, then there is clearly a lot of deadweight, but if the aim is something more grand – about saying that society shares the responsibility of bringing up children, then actually there probably isn’t any deadweight. Deadweight by its very definition depends on what your desired outcome is – in the case of EMA it is nice and easy because the govt told us from scratch it was participation – although I they widened that, rightly I think, to include attainment as well. Based on this proper definition of deadweight, can you think of any social programmes that have a greater degree of deadweight than EMA? Or even the same?

It’s just a number and you’ve arbitrarily said ‘that’s too much.’

Well yes, in the sense that 18 is an arbitrary cut off to vote, 17 is arbitrary to drive. You’ve got to have cut offs somewhere.  Personally I think you should seek to reduce any element of deadweight and then if you can’t reduce it below a certain point – I’d go for about a half, which you’re right is arbitrary – then you have to consider the value of the programme itself. Some programmes will have high deadweight that can’t be reduced that are still worthwhile. I cannot think of any at the moment, but I am prepared to believe there are some. EMA isn’t one of them.

But the logic of your position is the culling of many critical programmes that I’m sure do work in your view.

Such as? Give me one example of a genuine case of +50% deadweight, and chances are I think you could either make savings or cut. There might be one or two exceptions, or you might come up with your dreadful examples of GP check ups again, but overall, I think I’d cut.

For some reason, you’ve jus decided to single out EMA (perhaps because it’s a programme that the coalition wants to cut?)

I think every social program has to have its costs and benefits appraised. I think there is definitely a case for cutting some of the programmes I mentioned above, but I would agree with Neil O’Brien that EMA is one of the least effective ones out there. The reason I have made such a fuss about it is partly because I was absolutely baffled by the passionate defences it has received, but also because, in a very human and one-eyed sense, it affects me. You’re right, there are other programmes that are as bad  (although not nearly as many as you seem to think) Giving winter fuel to Fred Goodwin is just as outrageous, but I don’t have to deal with the consequences of that every weekday. I expect if Fred Goodwin and Polly Toynbee sat at my work every day whinging about losing their winter fuel, I’d start to get pretty aggravated about that too. The reason why I am so het up about EMA is because, as a teacher, it affects me. I am amazed by the fact that every teacher I know, regardless of their political leanings, thinks EMA is a bit of a racket, and yet the defence of it in the media has been so impassioned. I posted something like this on CiF recently and one respondent went out of her way to say how wrong and unfair and evil she thought the Lib Dems were, and she wouldn’t want anyone to think she liked me or supported me, but on this particular issue...she thought I was right.  Unsurprisingly she was a teacher.

PART 3

My final point is an observation. Throughout your analysis you have taken a very school-centric view. This is problematic in a number of ways. Though I haven’t got access to the numbers, given the nature of the cohort we are talking about, a larger proportion will actually be in college rather than school. This creates issues for your argument in a number of ways. 

Firstly, College qualifications are measured by percentage achievement at various levels rather than UCAS tariff points.

You’re right I haven’t taught in a college, but what do you mean by ‘college qualifications’? Do you mean NVQs? They are the only significant qualification offered in a college that aren’t included on the UCAS tariff - and in fact the NVQ in accounting is. The last time I looked the bulk of college qualifications were included on the UCAS tariff, but I am happy for you to tell me otherwise.. That is the whole point of the tariff – to provide equivalences. If you look here, you’ll see how many qualifications the tariff includes.  For all of them, the average 2.55 points improvement EMA brings is tiny, and even more tiny for vocational courses than academic ones in a lot of cases.

If say, an intervention increases success by 7% as seems plausible from the IFS analysis, this is enormous. A college that increases its achievement rates by that amount could go from being in the bottom to top quartile! (Incidentally, had the IFS used success rates rather than achievement rates, drop outs could have been factored in: the rate is calculated by achievement x retention. This doesn’t mean that scaling up of achievements is a problem that can’t be surmounted which you to seem to suggest- in fact, p.7 of the IFS report states explicitly that they can be, albeit in a rough and ready fashion.) I would encourage looking at the by college evidence presented in this article in TES: http://www.tes.co.uk/article.aspx?storycode=6064883

This is another classic sleight of hand article - most of the figures it cites are about kids staying on and ‘completing’, which is a lovely euphemism. It does state that there was a better pass rate, but how much better and what the average pass rate was isn’t mentioned, and nor is the real world use of the qualification – ie, if it has led to them getting away from NEETdom or just postponing it.

Secondly, your assertion- and that is what it is- that headmasters can allocate the fund efficiently may be so (though I some scepticism- it opens a scope for gaming of the system where students claim that they would not continue/ drop out unless they receive the payment but the EMA suffers from gaming also in fairness.) However, move into a College scenario where you may have 2000 students (Lambeth has 2,078.)

This is all you need to know about Labour’s love of centralisation. A headteacher in charge of 2000 kids won’t be able to do a better job of allocating resources than a central bureaucracy in charge of 600,000? Yeah, all right, that’s an ‘assertion’, of course there is no evidence for it.

I’m afraid your optimism re efficient allocation becomes impossible in such a scenario (Lambeth College will have 1000 students on EMA!) I should state that in the LFF piece I stated clearly that £50million was better than nothing and £100million would be even better still! 

What all this means in practice is that this policy is not a matter of ‘spending £500million to get £50million benefit.’ The only thing to back this up is your ‘deadweight’ argument which, as I have shown, is a nonsense.

No you haven’t

Actually, the overall benefit is significantly greater than that. The cost to the taxpayer of a NEET is £56,000 over their lifetime. So if just 9,000 end up as NEETs as a result (hands up, I doubled the expenditure over two years in the LFF piece but should have also divided it by two realistically given it’s a two year cohort which meant I quoted 18,000 instead of a more realistic 9,000- silly me to hinder my case in that way!) of the cut then the policy would pay for itself in fiscal terms (and we haven’t even begun to discuss the individual and social benefits….) Given a youth unemployment rate of somewhere in the region of 20% it will be more than 9,000 who become NEET.

Obviously all this assumes that kids getting EMA won’t become NEETs when they have finished. It may sound depressing but the facts and my experience suggest that for a lot of kids, EMA merely postpones NEETdom rather than abolishing it.

So if your argument is that we don’t want to pay now to save later and are willing to take the social costs etc. on the chin over and above that then fine. By why all this statistical dancing to get to that point? I say invest now, save later, unleash individual opportunity, and minimise negative social impacts of low or no qualifications and unemployment. Or don’t. That’s the choice.

You forget as well the problem of opportunity costs. Suppose we could say that EMA did have a minimal and overall positive impact. If we could spend that money anywhere else in education and get a greater return, then EMA is still a failure. One of the other things the IFS report shows is that the reason EMA didn’t have an impact on a lot of NEETs was because their real barrier to achievement wasn’t money, but prior achievement. So therefore a programme which put money into improving attainment at 11-16 could potentially have a better return on investment than EMA. I discuss the problem of returns on investment further here.

And one final thing – the reason why I’ve always felt EMA needed an exceptionally high success rate was because of the significant moral qualms I have about the concept. What’s your view on these?


UPDATE: Just want to say that I do appreciate you taking the time to post on this and at no point have I intended to be rude or patronising, so apologies if it comes across like that. 

Not chucking printers at someone = being exceptionally close to them

A friend sent me this article by Richard Grayson.

Basically, Grayson, former Lib Dem director of policy, argues that the Lib Dems should start thinking about a coalition with Labour. Nothing wrong with that in principle, but along the way he makes some fairly startling assertions.

• Grayson suggests that Lib Dems disillusioned over tuition fees could find a home in the Labour party. This is absurd. Labour introduced tuition fees, top-up fees and commissioned the Browne report. Does anybody seriously think if they had won the election they wouldn’t have introduced a similar system – and indeed probably one that was less fair?
• Labour haven’t put any concrete alternative on the table. Their leader and Shadow Chancellor can barely agree on a policy. And the one Ed M proposes was specifically rejected and ruled out by Labour in government.
• ‘Liberal Democrat leadership exceptionally close to the Conservative leadership’ – what a remarkably silly assertion. Does Dr Grayson not understand the concept of coalition government? Our leadership (and of course our membership too at the Birmingham Special Conference) and Tory leadership have agreed the coalition programme and are delivering it. This means they have to work together in a constructive fashion in the best interests of the country – hardly unreasonable. But I suppose after three years of a bloke whose definition of 'working together constructively' involved chucking printers around the office, civil co-operation probably will get mistaken for being 'exceptionally close'.  And after 13 years of the Blair-Brown saga the idea of people from the same party working together is pretty novel, let alone people from different parties. 
• ‘Liberal Democrat members have more in common with Labour and Green members than we do with our own leadership.’ Really? I certainly don’t have very much at all in common than Labour members - the vast majority of whom supported the Labour whilst it went to war in Iraq, introduced ID cards, tried to introduced 90 days detention without trial, massively increased the use of PFI, flunked the chance to scrap the House of Lords and introduce PR, locked up immigrant kids and dumbed down the education system.  And I couldn’t be further from the type of people who ran Labour’s campaign in Oldham at the last General Election and the many (although I accept not all) Labour MPs and members that have defended him. If Dr Grayson feel he has more in common with them than Vince Cable, Chris Huhne and Nick Clegg that’s his choice – but I fear that says more about him than it does about our leadership.  
• Dr Grayson then goes on to argue that Ed M is a pluralist politician. Again, I think the three years of Gordon Brown have really lowered the bar for what you have to do to be considered pluralist.  He must have missed Ed’s expressed desire to make the Liberal Democrats ‘extinct’ and conveniently forgotten that Ed M appointed Phil Woolas to his front bench team – as Immigration Minister no less – despite his appalling campaign against the Liberal Democrats in Oldham. He has consistently attacked the Liberal Democrats at every turn. That is his right, but it is hardly the action of a pluralist politician. 
• And Dr Grayson’s main piece of evidence that Ed is indeed a pluralist? The fact that he is supporting AV. OH MY GOD – a Labour politician supporting a policy that was in the manifesto he personally wrote counts as radical pluralism, words fail me. 
• For Dr Grayson to attack Nick Clegg for quoting JS Mill too much is rather odd as well. JS Mill’s ‘On Liberty’ is the text passed from President to President of our party for good reason – it is the defining text of Liberalism.  Dr Grayson also has a rather alarming lack of understanding of Mill. Mill himself wasn’t a man stuck in ‘1850s liberalism,’ he looks well beyond the traditional liberalism of his day, especially in his later years. He was campaigning for women to have the votes against socialists and trade unionists who wanted to keep women quiet in the home. Not only that, but I can think of no intellectual whose views I would rather have determining policy at the moment. A dose of Millite liberalism is what this country sorely needs after 13 years of social and economic authoritarianism from Labour.   I would suggest Dr Grayson might benefit from reading Richard Reeve’s rather excellent biography of the great man.  
• There is nothing wrong with us engaging with the Labour party in principle. But we are an independent political party, not merely Labour’s little brother. And if they want to co-operate with us then should start by apologising for all their deeply illiberal measures they took in government - and developing some policies that are a bit more liberal.

No stats - the moral case for abolishing EMA

I have blogged a lot about the economical and statistical effects of EMA, but it's as well to point out that this is not my only objection to EMA.

Scroll to just over four minutes in to hear what he has to say:

"We're from the slums of London, yeah, how do they expect us to pay nine thousand for uni fees and EMA, EMA, the only thing keeping us in college, what's stopping us from doing drug deals on the street anymore, nothing."

EMA started as a government bribe to kids. We shouldn't be surprised that the same kids are now using it to blackmail the government.

I've spoken about the attitude EMA creates before, with reference to the Save EMA website. This kid is clearly just like some of the ones I quoted there - viewing himself as a complete victim of circumstance, with no idea of his own individual responsibility or agency.

I don't want to get extreme about this. Clearly environment does have an impact on upbringing. But there is a very definite line between saying 'these proposals might lead to a slight increase in the number of kids doing drugs' and saying 'these proposals mean I will go out there and become a drug dealer'.

And the EMA protesters are crossing that line. Again, and again, and again.

You have a choice. You always have a choice, and in modern Britain you have very many choices, probably more than 95% of people who are currently alive or have ever been alive.

Shamima Blake, who I talked about a few weeks ago, is from East London too (I'm going to assume that's what he meant by the "slums of London"). But she chose to read the proposals, to listen critically to the scaremongers, and to work hard to get to university.

This kid, however, looks like he is going to choose to be a drug dealer. It wouldn't surprise me if in a year's time he's up in front of the bench and tells the judge 'it's not my fault right, it's not my fault, if they'd had EMA I would have been a brain surgeon by now!'

The sad thing is, there's people out there who will believe him.

What Labour think is cost-effective

Anthony Painter has responded to my post here defending EMA and I have responded here because my post wouldn't fit in the comments box.  I’m assuming that he is this Anthony Painter, prospective Labour MEP. He’s the first response I’ve had that hasn’t either attacked me for being a Lib Dem and has actually engaged with the substantial issues, so thank you.  We’re having a statistical quibble about some points – he doesn’t appear to have read to the end of the IFS report,  which does make a difference, see here for even further details – but what is interesting is that he completely concedes the deadweight issue – the problem that 90% of EMA goes to kids who would stay on without it.  NUS, are you reading this? Numerate Labour supporter concedes deadweight issue.

Then, incredibly, he does two things. After acknowledging that 90% of EMA is wasted in this way, he says 1) that it would be impossible to have a better targeted scheme with similar aims and 2) given this, EMA in its current form should be protected – it still offers value for money.

Quite frankly, this tells you all you need to know about the current Labour party and all you need to know about why there is such a fiscal mess. First, there’s the astonishing assertion that in a scheme where all sensible people accept that 90% of the cost is wasted, you cannot possibly cut any of that waste.  This should at first sight be so implausible as to be ridiculous. Imagine if teachers only worked for 10% of the school day. Imagine if you bought 100 interactive whiteboards and only ten worked. Imagine if you built ten prisons but only used one of them. Even the EMA-defender of this paper acknowledges that there are quite a few cuts you could make to solve the deadweight issue – not as many as the government are proposing, certainly, but still quite a few. But Anthony Painter doesn’t want to even consider that – for him, it’s a  ‘choice is between a £500 million policy which meets its objectives or a £50 million scheme that doesn't’.

Let’s just as a thought experiment accept that he is right here and that there absolutely nothing you could do about the £450m deadweight cost. I am afraid if this was the case and there was only the ‘choice’ presented above I still could not defend EMA.  You are getting an improvement in 5-10% of the kids on EMA, and to get that improvement you have to give away money to another 90-95%. That seems to me colossally poor value for money. I can think of several ways of spending £500m of the education budget that would be much more valuable. Of course, what people like Anthony Painter then turn round and say is that I am condemning thousands of kids to be drop outs. Where would this end?  Doubtless if you gave every NEET a million quid on pain of attending college three times a week they wouldn’t be NEETs any more, does that mean my failure to advocate such a policy is condemning thousands of kids to be NEETs? If you paid every MP a salary of £50 million quid, I am sure that most of them wouldn’t fiddle their expenses, does that mean my failure to advocate such a policy is condemning MPs to fiddle expenses? Quite apart from the astonishing elision of individual moral responsibility in this vision, it’s just not affordable. I’ve got a lot of sympathy here with the people on NICE and Sarah Palin’s beloved ‘death panels’. There are some ‘solutions’ to medical and social problems that offer a very limited benefit for an extremely large cost.  It sounds callous to say you want to deny people these benefits, but as Paul Krugman – no crazy right-winger he – points out, unless you do so, you get rampant healthcare inflation in the US example, and social policy inflation in the UK example. It's the dartboard theory of public finances.

But in actual fact, in this example the above paragraph is actually entirely academic because if you are wasting 90% of a budget there are quite clearly ways to make savings, and any political party who tell you otherwise shouldn’t be let near the public finances. 

Response to Anthony Painter

I wanted to put this in the comments page here but it was too long.

In summary:

Essentially, for all your quibbles, you agree with my statistics but quibble about the analysis. You agree that EMA positively impacts 5-10% of students and is wasted on the rest. I hope the NUS are reading this.

I have never claimed that EMA doesn’t benefit anyone. It would be odd indeed if a £550million programme didn’t have some benefits. My claim has always been that the benefits EMA brings are out of all proportion to its costs. My claim has always been that, based on my experience and all the data in these papers:

• ·         About 90% of EMA goes to kids who don’t need it and who would be at college anyway.
• ·         About 5% goes to kids who would drop out without it but who don’t get any significant benefit from being at college – either dropping out or not getting any qualifications.
• ·         The other 5% goes to kids who do benefit from it.

That’s the argument I have always made and I see nothing from any of your quibbles to change this.

Where we differ is that you think it’s a) impossible to improve that waste rate and b) that the programme is still worth it. I differ on both these. The IFS – and indeed the CBfT, in their pro-EMA report – suggest a number of ways you could more tightly target the allowance and the government plan to give it to schools will clearly eliminate a lot of waste. If it truly proved impossible to eliminate the waste, then I am afraid EMA would not be worth it.

OK, now the detail. Let’s take this bit by bit.

The difference between the area and individual impact of EMA.

The figures you mainly quote above are pilot area averages (thou you do jump about a bit).

If by jumping about a bit you mean I have read the entire paper...

It is important to read pp 6-7 of the IFS report to understand why this is problematic. The key point is:

"We therefore suggest as a rule of thumb that, in order to obtain the effect of the EMA across those who received it, the estimates above be multiplied by a factor of 2½ (for outcomes at age 16) or 3 (for participation at age 17 and attainment outcomes)."So the participation and attainment impacts are in the main much greater than you suggest.

It is important to read appendix D of the report to understand they’ve already grossed up the figures. The stats I take from the tables are ones that have already had this adjustment made to them – ‘The headline results (presented in Tables 1a and 1b) can therefore be ‘grossed up’ to give the effect of the EMA on those who received it by dividing the estimated impacts by the proportions above... doing so yields the notional figures that are presented in Table 1 alongside the actual estimated impacts.’ The 7.3 percentage point figure for participation I have been quoting all along is the one that has already been ‘grossed up’. Not only that, but when they broke it down by age, sex, method, etc., I picked the largest figure, the one which was most damaging to my case. Also, after reconsidering this, I have realised that in one of my previous posts, I assumed that 7.3% figure wasn’t grossed up, and so I grossed it up again! So in this post, and in the letter to the NUS here I am actually wrong – EMA has even LESS of an impact than I said! See here for details.

As for the impact on attainment, you are right in that I and the IFS do not gross up the impacts on attainment. This is for two reasons. Firstly, with attainment more than with participation you want to get a picture of the impact the policy has had on the entire cohort. This is because the base points rate is also measured over the entire cohort. It’s practically impossible to pick a base which is the average of kids on EMA who would have been doing their courses without it. You don’t know which kids and which results belong to those who would have been there anyway, and those who have been incentivised to stay on. As I pointed out in my original post, the IFS are aware of this problem with the base rates, and that's why I think they do gross up participation and don't gross up attainment. I do think that the attainment results are probably the most unreliable part of this survey - there is this problem with the base rate, and the statistical significances aren't very good either - when you factor in the problem of real world significance I talked about last time I would not be inclined to draw anything meaningful from them.

Secondly, if you try and gross up the attainment stats, you are inevitably have to pick stats which are self selecting. Let me explain: you can obviously only calculate attainment by looking at 18/19 year olds. But that means you exclude all the 16 year olds who drop out before doing any qualifications. The only way you can measure attainment of EMA recipients is to exclude all the EMA recipients who didn’t attain anything – which is quite a lot and so will distort your figures. 25% of 17 year olds getting EMA don’t progress to getting it at 18 – that’s very significant.

Also, so paltry are the attainment rates that you actually could gross them up by a factor of three and they’d still be educationally insignificant – still well within the margin of error of the exam boards.

2. The results of the EMA on black and Asian students (that you applaud and accept to be significant)

I didn’t go as far as applauding them – they’re actually extremely paltry, just slightly less paltry than the ones for white kids.I said they were the only ones that were educationally significant – they are, just.

are not just an anomaly- they indicate the real impact of the policy. These communities are more likely to be recipients of EMA.

Ah, classic sleight of hand. Yes, a greater percentage of the ethnic minority community are likely to be recipients of EMA than the percentage of the white community who are recipients; but most EMA recipients are still white because most of the UK is white! Ethnic minorities are still a small minority of overall recipients (about 11% based on the figures in this survey); they’re just a slightly bigger minority of EMA recipients than they are of the entire population.  So again, most of EMA is wasted. Also, I would suggest that the fact that we can so obviously tell that it’s ethnic minorities who benefit from it makes a targeted scheme like the one outlined below easier to administer – you’d target the money to areas with high ethnic minorities, and/or make principals aware of this – although the problem with this is the problem of ‘reverse racism’. Do you really think that it’s insignificant that EMA has no impact on attainment on the largest ethnic group?

It should be emphasised however that even here the impacts are underestimates- you have to be enrolled and participating in order to receive EMA (as well as meeting the income criteria.)

I’ve dealt with this above. And it should also be emphasised that the statistical significance of a lot of them isn’t very high either – some are only at the 10% level of certainty.

3. You talk about A Levels but largely neglect level 3 qualifications where the impact is greater. There is a reason for this. Colleges use EMA as a 'nudge'- receipt is dependent on attendance and punctuality. Without EMA, there's no nudge and grades decline. Vocational level 3 qualifications are entry level in a whole range of careers....

If you’d read the post properly I deal with this issue. The IFS themselves are extremely unclear with the terms. The bit you’ve obviously missed is: there seems to be quite a bit of uncertainty about the points tariff the researchers talk about. At one point they refer to it as the ''Key Stage Five tariff'. It's a common misconception even amongst educators, but there is actually no such thing as Key Stage Five - at least not in the official sense that there is with Key Stages Three and Four.  Then, at other times, whilst appearing to refer to the same thing, they talk about the 'A-level tariff'. Of course, there is such a thing as A-levels, but I don’t think there is such thing as an exclusive 'A-level tariff'.  I have assumed what they are talking about here is the UCAS tariff, especially as they use the phrases ‘Key Stage Five tariff’ and ’A-level tariff’ interchangeably. The UCAS tariff includes a whole range of different qualifications, academic and vocational. So when I have been referring here to A-levels, don't assume that I am excluding all the kids doing vocational courses. The tariff includes them, and believe me, an increase of 2.5 points is just as unimpressive for BTec nationals as it is for A-levels - arguably even more unimpressive.

An improvement of 2.5 tariff points in a BTec national diploma is even less educationally significant than the same across three A-levels.

4. Your deadweight argument fails as you do not subject alternative schemes to the same level of scrutiny. The coalition's scheme has a fundamental weakness. You can't identify those who would drop out without EMA.

Given that it seems to impact ethnic minorities the most, then you very clearly can – although I accept that this brings up the issue of race discrimination. Be interested to see what you think about this. Furthermore, the IFS themselves saw their survey not just as an overall impact evaluation, but as ‘being able to provide more breakdowns by subgroups which will shed light on which groups the EMA is having a relatively large and a relatively small impact upon.’ There’s a mine of information in there about the types of kids EMA benefits, not just according to race but according to gender, previous performance and wealth.  So to say ‘you can’t identify those who would drop out without EMA’ also seems to suggest you stopped reading the IFS report at page 7. Even the CfBT, who wrote a report defending EMA, acknowledge that you could probably cut the 10 and 20 quid bands. But you won’t even concede that!

This creates a moral hazard. Students have an incentive to threaten to drop out or to threaten to not enrol in order to force the institution's hands. How do you judge the genuine case? So the bureaucracy argument is neither here nor there- a large number of people who are hit by EMA will inevitably be missed by a discretionary scheme. The choice is between a £500 million policy which meets its objectives or a £50 million scheme that doesn't with the knock-on social, economic and fiscal impacts of swelling the realms of the NEETs.

 I think this is the area where I disagree with you most fundamentally, and not for reasons of statistics. Perhaps some kids will drop out even though they could stay on just in an attempt to ’bluff’ the institution. That much is apparent from the comments on the EMA website and this lovely chap here– but I would suggest any kid going through life with that attitude is going to need a lot more than EMA to be successful.  We need to remind kids that six years ago EMA didn't exist and poor kids still went to college then. In the end I think most kids will get this - when you talk to most of mine they do realise this and admit that, as much as they like it, they would stay on without EMA. Plus, I have a greater faith in principals and schools to know who really needs the money, as opposed to a more distant means test which we all know was open to tons of abuses. Kids are actually far less likely to game the system when they see the people responsible for giving them the money every day. Plus, this current 500 million scheme does not meet its objectives – one of its objectives was to reduce the NEET rate, which it hasn’t done. The IFS in another paper note that much of what increase there was in the staying on rate came from kids who would otherwise have been in jobs, not kids who would otherwise have been NEETs.  That’s why bringing it in didn’t have much of an impact on NEETs, and getting rid of it won’t either.

So for all the statistical dancing around here: your numbers are flawed and misinterpret the evidence;

Not at all, unlike you I read the whole report and got to Appendix D where they explain they’ve already grossed up the figures

you fail to appreciate the nudge aspect of the policy and how this is most particularly useful in a College environment; and fails to properly scrutinise the alternative scenario next to the default. And if it's abolished 10,000s more students will be condemned to failure.

Again, this is another attitude I have a problem with. Students will not be ‘condemned to failure’ by this. That suggests students are passive victims with no agency of their own. Do you really want to reduce poor kids to this level? Ed Miliband himself has said one of the problems with markets and the state is that they both tend to view people not as individuals but as statistics. I think that is exactly what you are doing here.  Also, you basically seem to accept my point that there is a lot of waste. Your argument is that there is absolutely no way we could cut that waste. I find that very strange. I can’t think of any area of life where we couldn’t improve a 90% waste rate. What would you say if we equipped every new classroom with interactive whiteboards that only worked 90% of the time? What would you say if we employed teachers who only worked for 10% of the school day? What if we bought tables and chairs that could only be used for 10% of the school day? However, even if we accept your point that there is no way we could target the scheme better, you seem to think it is fine that we give 450million to kids who don’t need it  as long as 50 million reaches the kids who do.  If that were the case – and it isn’t – I am afraid I would have to abolish the scheme entirely.  There are so very, very, very many more ways you could spend £500 million in education and get a better return on investment.  Even in an era without tight budget restraints, I think the waste of 450million would be unjustifiable.  In an era with them, it’s ludicrous.

Oh, and finally...the numbers of those affected are far greater than you argue. You state that the policy helps ’12,000-18,000’ kids. This is completely wrong. In 2009-10 there were 643,000 recipients. According to your statistic of 83% deadweight loss, that means 109,310 will be positively impacted. Using the survey figure of 10% retention, it’s 64,300. This is a two-year cohort so very quickly over say a ten year period you are looking at 300,000 or so to 550,000 impacted. In anybody’s world that’s significant.

Thanks to you kindly pointing out the error in the double grossing up done by CbFT, the stat is actually 95% deadweight (see here).That also chimes with the RCU survey. So that’s two surveys, including the most rigorous,  which suggest 5-6% and a couple – with less rigorous methodology – which suggest 10-15%. Let’s split the difference at 8%. So that’s 51440  potential drop outs. Bear in mind, as I have pointed out, that not all of them would have gone on to complete their courses or get any qualifications anyway – that is, for from preventing drop outs, EMA probably just delayed some and may have even caused others. We can probably say on the evidence that about a quarter of those 51k would have dropped out anyway, so now we are down to 38k affected.  And as much as you may have doubts about the 50m fund, you have to concede that it will make some difference, don’t you? It will probably have some deadweight too, but it’ll be miles less than the EMA – in fact, were I a head teacher, I would feel pretty confident about getting a 10% deadweight rate. So we can assume 45m will be spread across the most needy 38k students – 1185 a year, about the same as the current maximum EMA of c.1170. Easy.Of course, you can never guarantee that not absolutely one individual will suffer as a result of something, but according to all the evidence, we can feel pretty certain it won’t cause significant problems.

I also think it's very interesting that based on a clearly unfinished reading of the IFS report, you saw fit to take me to task over what was a fairly complex statistical issue - one that I was right about anyway, and that had I been wrong would have made a fairly small difference. I notice you claim on Twitter that this makes my post 'deeply flawed'. Yet you don't feel inclined to comment on my post about the NUS's misrepresentation of EMA, which is based on a misunderstanding of percentages and percentage points - fairly elementary I would like to think.  This error means they increase the drop out rate from 17% (which as I now show here was already incorrect) to 65%.

If you thought my alleged error was deeply flawed, what would you call that?  Very deeply flawed? Deeply deeply deeply flawed? As, unlike the NUS, you clearly are numerate, I'm interested to know.

Even if you cannot agree that spending 500million pounds to get a 50 million impact is terrible value for money, can you at least agree that supporters of EMA should base their campaigns on the facts? Will you be joining me in asking the NUS to correct these errors?

Update: I was wrong. EMA is EVEN WORSE than I thought...

I was wrong – EMA is even more of a waste than I thought.

In this post on the issue of EMA deadweight, I analysed a report by CBfT which the NUS had cited wrongly.

The NUS had cited it wrongly – but what I didn’t notice was that the CBfT had themselves cited research from the IFS wrongly.

Essentially, what has happened is that the IFS calculated what impact EMA had on participation on kids living in an area. Then, in order to get the impact on kids who were actually receiving EMA as opposed to just living in the area, they had to gross up their original impact figure to take into account that not every kid living in an area got EMA. But they include their grossed up figures in the main body of the report.

The CfBT make the mistake of assuming these were the ungrossed up figures, and therefore gross up again. When reading the CfBT report, I was a bit baffled by that, but it’s only now that I’ve worked it out. Essentially it means the CfBT overestimate the impact of EMA by about 2.75.  Hilariously, even with this double grossing up EMA still wasn’t that effective and the CfBT, who defend EMA, still had to admit that the deadweight cost ‘may be thought to be too high.’

Let’s just get this clear because it is quite funny.  The CfBT are defending EMA. They do their stats, slip up, gross up twice and overestimate impact of EMA by about 2.75.

Even after overestimating the impact of EMA in this way, they still have to admit the deadweight is huge.

And they are supporters of EMA!!

Here’s the full stat breakdown.

The CbFT report, written by Mick Fletcher, used the IFS research to claim:

The most recent figures show that 43% of full-time students aged 17-18 receive EMAs. Taking an average of recent evaluations EMA may have increased participation by up to 7 percentage points, leaving some 36 percentage points as deadweight.

As I showed, those 36 percentage points equal 83 percent.  However, to get this you have to accept the assertion that ‘taking an average of recent evaluations EMA may have increased participation by up to 7 percentage points.’ I assumed this 7% figure was taken from the IFS study which Fletcher cites earlier on. However, when I reread Fletcher’s rationale for using the 7% figure instead of some of the other ones used in the IFS report, I realised he’d used the wrong figure for this particular calculation.

The IFS report comes up with sixteen different figures for EMA’s impact on participation. It calculates it using two methods – comparing pilot areas to control areas, and comparing all pilots to all of the rest of England. Then, it calculates  the effect on the entire cohort and the effect just on the cohort who actually took up EMA. Then, it calculates each effect for 16 year olds and for 17 year olds. And finally, it calculates each by sex.  It would have been nice if they had come up with a figure for both sexes and both ages as this might have helped give a better overview. Anyway, this is what we have.

Now, I am not particularly concerned about which method, sex or age you choose. What matters for this particular thing we want the data for is the bit in the left hand column – whether we look at the impact across the pilot areas as a whole, or the impact across EMA recipients. The former measure produces much lower figures because of course, you are measuring the increase in participation in relation to the entire cohort. For the second measure, you are measuring the increase in participation only amongst EMA recipients. So you are obviously going to get higher impact rates. I assume what they mean here is that they attribute EMA to be responsible for x percentage points of those EMA recipients. So actually, this figure is exactly the one we want. We don’t then have to reduce it down in the method I show above, because they’ve already done that. If we were going to go through the measure I outlined above, then we should do it with the figure for the impact across the pilot areas as a whole. And if we do that, then we obviously end up with the same figure that’s in ‘impact across EMA recipients’ column, because of course this entire piece is based on the same data.

So, when I wrote this in my previous post:

We have 100 sixth form students.  35.7 get EMA but would be at sixth form anyway. 7.3 get EMA and wouldn’t be in the sixth form without it.  The other 57 are at sixth form and don’t get the cash. Of the 43 kids who get EMA, it’s only 7.3 kids who are actually being incentivised by it to stay on at 6th form. The other 35.7 would be there anyway. Thus, the deadweight cost is 35.7/43, or 83 %

Actually I should have used the following figures:

We have 100 sixth form students.  40.1 (43-2.9) get EMA but would be at sixth form anyway. 2.9 get EMA and wouldn’t be in the sixth form without it.  The other 57 are at sixth form and don’t get the cash. Of the 43 kids who get EMA, it’s only 2.9 kids who are actually being incentivised by it to stay on at 6th form. The other 40.1 would be there anyway. Thus, the deadweight cost is 40.1/43, or 93.3 %.

Thus,  only 6.7 percentage points of pupils already receiving EMA are said by this data to have been encouraged to stay on as a result of EMA. The reason why this isn’t exactly the same as the 7.3 it should be to match up with the table above is because Fletcher uses the figure of 43% EMA take up whereas the IFS have the exact ones for the cohorts they are discussing which are 40% for 16yr olds and 30% for 17 yr olds. It’s all in appendix D.

So, if we want to calculate the number of people on EMA who are only carrying on in further ed because of EMA, then it doesn’t matter which method we use or whether we look at 16 year olds, 17 year olds, males or females. What matters is that we use the SECOND row for each of these segments. 

That gives us 5.5%, 7,3%, 2%, 5.5%, 6.3%, 8.1%, 0.6% and 4.5%. It isn’t strictly accurate to take a simple average of these, because there will probably be more females than males and definitely more 16 year olds than 17 year olds, but let’s do so anyway to get a rough idea: 4.975%. So, according to the IFS, the most reputable organisation out there, 5% of EMA recipients would not be there if they weren’t getting the money.  That means that 30,000 kids are in college who wouldn’t have been otherwise (and this tells us nothing about their drop out rate or attainment rate), and we have given 570,000 kids £500 million quid they didn't need.

But apparently, making savings would be impossible.

I'd hate to see what Labour thought a waste of money was.