The IFS on EMA, again, courtesy of Channel Four

I am writing this early Tuesday morning. There is an interesting article on the Channel Four blog which suggests that later today the IFS are going to publish a paper suggesting that the costs of EMA are more than recouped.  Channel Four’s blog is the only one to feature this, and from the article it seems as if the details of the paper have been leaked by the report’s author, Haroon Chowdry, to the Channel Four blogger, Faisal Islam.

I’ll deal with what we learn about this report in a minute, but first of all let’s take Islam’s comments on it.  I’d be interested to know how many of his fairly partisan comments are shared by the author of the paper who seems to have leaked it, and how many are his own. Anyway, that aside, he takes an extremely negative tone towards the government that in some cases are absolutely not justified by the facts and in others, well, we’ll have to wait and see what the report actually says.

So, Islam’s comments. He says that

 ‘the Institute for Fiscal Studies will tomorrow fire a flaming bolt through the middle of some rather second rate analysis put out by the Government in favour of scrapping this policy.’ 

Which second rate analysis would this be? The analysis by Policy Exchange of the effectiveness of EMA, which cited analysis by...the IFS? The Policy Exchange report was written by Sam Freedman, who is now Michael Gove’s policy adviser, so it seems fair to say the government are very well aware of the first-rate analysis done by the IFS on this issue - which backs up their ideas.

Then Islam says this:

That is indeed the finding of the only peer-reviewed analysis of the EMA, done by the IFS team, but sadly unavailable online as it is an article for the Journal of Human Resources (hat tip to the FT’s Christopher Cook). Tomorrow’s IFS analysis is likely to reaffirm that finding.

Hmm. I automatically assumed that all the IFS’s reports were peer-reviewed, in which case there are a whole library of them here, freely available, on their website. Which 'only peer-reviewed analysis' is Islam talking about? Perhaps I’m wrong and these ones on the IFS website are not peer-reviewed.  The analysis I have leant on when doing my posts on this blog has been this one here, which as far as I know is the most recent and the only one to take attainment into account, not just participation.

Unfortunately, I am going to be busy at work for most of the rest of this week, so I am not going to be able to read this report which is published tomorrow. I’ll just flag up a few issues I have with it based on what Islam quotes Chowdry as saying. I am not seeking to question any of the IFS’s statistics – as far as I know anything about them their intellectual credibility is beyond doubt, which is why I used their stats when making my blog posts on this.  But as my discussions with Anthony Painter have shown, stats are only one aspect of a case. Good people should be able to agree on stats, and give or take a few salient points I think Painter and I largely do. Once the stats have been established, that’s actually when the interesting stuff starts. Anthony Painter thinks that a deadweight cost of 90% is fine, I don’t.  Our differences are, to use a word much in vogue of late, “ideological”. That is, we’re both ideological, not, as Labour would want you to believe, that they are the staunch upholders of evidence-based rational politics and everyone else wants to change things according to silly ideas of morals and things.  None of this is to argue for a second that the stats don’t matter.  Of course they will affect your understanding of an issue. That was why I was so keen to correct the NUS on their ridiculous error here. Many more people will think that EMA is worth it if they think the NUS’s stats are right than if they think Anthony Painter’s stats are right. If you say that 65% of kids will drop out if EMA is pulled (because, unbelievably, despite claiming to represent students, you can't tell the difference between a percentage and percentage point), then many more people will support it than if you say that 10% will. The stats do matter, but even when they have been established, that will not solve every disagreement, and nor should it.

So, the IFS are important in this argument because they can provide unimpeachable stats. But as I say, those stats can and will be interpreted in different ways, as I will show below. As well as this, I, and my fellow teachers, are important in this argument, because we can provide something the IFS cannot – evidence of the moral impact of EMA, which I deal with not in this post but here and here. Now, because of these moral qualms I have about it, then for me what counts as statistical 'success' of the EMA will probably be set a lot higher than for someone simply looking at the figures. Like I say, call me ideological.

To give an extreme example of what I mean by this, let's assume that I run a rigorous and controlled experiment that proves conclusively that shooting the kid who comes bottom in a test results in greater attainment in that test for the entire class. My facts and stats are unimpeachable, but it doesn't matter: no right thinking person would agree that this was a good policy. Its one 'cost' is greater than any possible benefit to attainment, however impressive. Now of course EMA isn't like shooting kids - like I say, this is an example. But I believe that EMA has many very significant drawbacks which aren't captured in the stats, so whilst I would be prepared to support it, for me the attainment/participation/future earnings/future wellbeing stats would have to be really, really good for me to do so. Plus, any programme, however successful, also has to justify its efficiency - could you achieve the same results for less? So even if we had a programme that cost, say, about 100,000 but instantly generated benefits worth ten times that, you could still argue that the programme should be reformed if you felt that only 90% of the money was getting you the benefit and you'd found a way to cut it that didn't affect the overall output.

OK, onto those facts:

“The initial outlay of the EMA policy is likely to be more than recouped by the increase in productivity that we expect to result from the 16- and 17-year-olds staying on in education for longer”. 

“is likely?” This is the first hint you get here that the actual findings of this report are not nearly as confident or secure as that 'flaming arrow' quote by Islam would have you believe. So the first thing I would want to do when I see the paper is to see how they’ve decided this.  I’d also be interested to see what the causal links are between 16 and 17 year olds staying on in school in longer, and an increase in their productivity.  This seems to me a bit like the reasoning that went around a few decades ago about degrees. People getting degrees earn more money, the reasoning went, so let’s encourage more people to do degrees and they will earn more money. More people now do degrees, and the graduate wage premium has fallen. Just because there may have been an increase in productivity associated with those who stayed on at school longer, does not mean that this will hold true if we increase the numbers who stay at school.

But it is a strong piece of evidence, that even if relatively small proportion stay on at school as a direct result of the EMA (and there are arguments about these numbers), it could still be a net benefit to the economy in the long run.

Note that 'could' again - not sure how reliable any of these statements are, again we have to wait for the report. This point seems to be accepting that EMA does only benefit a small proportion of those who get it. I find it particularly hard to believe in this case that the benefit to this small proportion is enough to eventually pay back the entire costs of EMA to everyone. The most generous IFS calculation I’ve seen of the deadweight cost is about 93%. Other less rigorous studies show 83%. Let’s be really generous and say it’s 80%. The increased earnings of 20% of kids are going to pay back the EMA costs of the entire 100%? That seems really implausible.  The cost of EMA is 550m a year, how long would it take for the net extra earnings of the 20% to reach that sum?  600k are on EMA, 20% of that is 120k, so if you could prove those 120k kids left college and instantly earnt 5k more than they would have done without, that would pay back the entire year’s costs of EMA and all their future increased earnings would be ‘profit’. But of course, it would only pay it back to the individuals, not to the Exchequer, as this post makes clear. So the taxpayer would still be out of pocket. Take a tax rate on that extra 5K of 33%, and it would take three years (five when you include the two where you were doling the cash out) before the scheme broke even, and then you would have to control for inflation over the five years since you started doling it out, so let's call it six years and that is before you start to think about kids going to uni - which might increase the wage premium but also delays it. All the tax on the kids’ extra earnings after that would be ‘profit’, or return on investment. Let’s assume that if the government invested the 550m in gold or bonds or used it to pay down debt or used it on another social programme – let’s say the govt could easily get 2% on that sum after inflation. In the six years it took EMA to break even you’d be well up with your alternative investment - compounded, you'd be 120m up by the time EMA broke even. And that’s on a wildly optimistic scheme that sees more kids benefitting from EMA than any study has shown and benefitting by a sum of money which I think is highly improbable – I don’t know any 18 year olds who earn 5k more than 16 yr olds by dint of the two years they did at college.  Now I don’t actually know that is what the report is saying, because of course I don’t have it. Perhaps it is saying that the increased earnings and productivity of 20% of kids will pay back the EMA costs of that 20%, which is an entirely different and far less impressive claim as it would still leave EMA overall costing an awful lot more than its benefits, and would push that notional break-even point I speak of above even further away, even on the most optimistic scheme.

Anyway, my essential point here is that EMA may very well give a long term return in its investment, in the same way that putting your money in a bank account paying 0.5 interest will give you a return.  It’s the scale of the return that matters. EMA may very well break even or turn a profit, but it has very significant opportunity costs – you could get a much better return on investment elsewhere.

There is of course a 'moral' or 'ideological' rejoinder to this that I am sympathetic to. Even if you could make 120m more elsewhere, isn't it worth 120m to give people a better job and better lifestyle? I am definitely sympathetic to this idea, which is why if I were reinvesting this money, I would not use it to pay down debt or to buy gold - I would use it in education, in a programme that could give you a much better short and long term return on investment - probably one involving primary or secondary, probably akin to...ooh...let me see...the Pupil Premium! And of course the other thing this report doesn't seem to take into account is that EMA isn't really being completely abolished - there's going to an extra 50 odd million in an emergency pot of money for needy kids. So any comparison would have to take into account the fact that at least some of this money will find its target - and probably a far greater percentage will find its target from this budget than from the EMA one.

One final thing:

As the IFS told me tonight: “This cost-benefit calculation does not include other benefits through other routes: for example, increases in educational attainment that might arise for the EMA recipients who would have stayed in education anyway, but now have more time to devote to studying.”

I can't believe the IFS have said this when their previous study specifically looked at the effect of EMA on the attainment of all kids receiving it. This is very baffling.As I have shown in a previous post, the increases in attainment achieved by kids on EMA are statistically significant, but not educationally significant - that is, I would be pretty confident of replicating them if you paid to send every exam paper off to be remarked. And of course any analysis of attainment is inevitably and unfortunately constrained by the fact that the English exam system is almost certainly not a reliable metric. I know this is an entirely separate argument, but it really does seem to me that the recent OECD figures go some way to proving the gut instinct a lot of people in education have that exams are getting easier.

Unfortunately I am going to be very busy for most of the rest of this week and probably won't have any time to read the paper until the weekend. I've written this quickly and whilst tired and it is all fairly tentative so there may unfortunately be some errors, but I hope it expresses the gist of my ideas.