Does EMA improve attainment?

Yesterday I published a rather long statistical analysis showing that most EMA money was going to kids who would have stayed on at college without it.  That is, EMA was having very little impact on the participation rate and was mostly going to kids who would have participated without it.

This is an important issue, because the stated main aim of EMA was to increase participation. Most of the analyses of its effectiveness have revolved around this issue, and it's often discussed as removing one of the 'barriers to participation.' So, the fact that it doesn't increase participation very much and that most of goes to kids who would have participated anyway are very significant facts.

However, despite the fact that I have just spent an inordinately long time examining the stats on this, it is not the only issue. In fact, to me, it is not even the main issue. It was always fairly obvious to me that if you pay not very well off 16 year olds to do something not very arduous that a lot of their mates are doing, you will get a lot more of them doing said thing. If you pay people to 'participate' in something, you will raise participation rates; the fact that large sums of taxpayers' money have been spent to prove what I thought was a fairly elementary economic proposition is slightly baffling. The one useful thing this research can tell us is the amount of increased participation, which as I have shown is fairly paltry.

However, to my mind participation is not the only point of EMA, and it's not the only point for two reasons.

First, EMA could have improved participation rates by 300% - that would not necessarily make it a success. Even if it had improved participation rates and you could prove it had, even if you could definitively prove that everyone receiving it would drop out without it, that would still not to me prove anything at all about EMA. To prove the success of EMA, the proof that it has increased participation is necessary, but not sufficient. You also need to prove attainment. This is because, funny enough, education is about more than just 'participation', just as jobs are about more than presenteeism.  Does EMA improve attainment? Does it increase what I think we have to wincingly call 'life chances'? Are the kids receiving it going on to university? To better jobs? To not be on the dole?  If all EMA does is get kids to sit quietly at the back of the class for two years before emerging with three Us at A-level, then it may have increased participation and got the unemployment numbers down, but it hasn't added anything to the lives or economic potential of the kids involved and it's merely delayed either them getting the same job they would have anyway, or them becoming a NEET.

This may sound an unduly pessimistic point, but this is what I see a lot of when I am teaching. I  have certainly taught kids on EMA who would have dropped out if they hadn't received EMA - and an awful lot of them really should have dropped out. They were only there for the money. They did poor work, only just made the attendance threshold to get their money, rarely did homework, rarely contributed in class, never met coursework deadlines and in general generated an awful lot of work for their teachers that would have been better spent on the kids who actually wanted to be there.  So what we really need the research to show is how many kids are there getting EMA who should be at college, who want to be at college, who will do well at college, and who will not be there unless they get their EMA?

Second, the reverse of this holds true. Even if EMA hadn't improved participation rates at all, even if all of it was going to pupils who would otherwise have stayed on in education, then it could still be a success if it had improved those students' grades. The main way it would have achieved this would have been in allowing students to stop paid work and devote more time to their studies. If this were the case, then I would have to concede that despite the 'deadweight' issue I outlined previously, it was still having some benefits. However, again, my anecdotal evidence suggested that pupils were not doing this. Lots of pupils on EMA continue in paid work. Even the NUS don't expect that students receiving it will give up paid work, merely that they will reduce their hours. And of course, it isn't necessarily the case that giving up or reducing paid work results in an increase to the time you devote to your studies.

Anyway, my anecdotes aside, let's consider the evidence. What impact does EMA have on attainment? Because the main aim of EMA was to increase participation, most of the surveys on it measure that. Only the most recent IFS survey, from what I can see, analyses attainment. This has been reported very positively, as improving attainment rates by 5-7%. Let's have a look at what it says.

 "Both males and females saw improvements in average A Level tariffs of roughly 4.5 percent of the base level." 

What this means is highly dependent on what the base level is, and the base level is different depending on the two different methods the IFS researchers use, and also depending on the way they break it down - by sex and by age.  This is not just hairsplitting. One of their methods defines the base level as "56.7, which we measure as the average Key Stage 5 tariff across all individuals in the pilot and control areas." 4.5% of 56.7 is an improvement of 2.5515 points. That would take you up to 59.2515, which is very close to one D grade but is unfortunately still just one E grade A-level. The improvement of 2.5515 points equates to ONE EIGHTH of ONE-A Level grade. All of the IFS's findings on average A level tariffs are a classic example of the difference between statistical significance and real-world significance. This is not something I have made up, it's something that is acknowledged by statisticians - here an introductory guide to statistics explains the difference:

"Many researchers get very excited when they have discovered a "statistically significant" finding, without really understanding what it means. When a statistic is significant, it simply means that you are very sure that the statistic is reliable. It doesn't mean the finding is important or that it has any decision-making utility. 

For example, suppose we give 1,000 people an IQ test, and we ask if there is a significant difference between male and female scores. The mean score for males is 98 and the mean score for females is 100. We use an independent groups t-test and find that the difference is significant at the .001 level. The big question is, "So what?". The difference between 98 and 100 on an IQ test is a very small difference...so small, in fact, that it’s not even important." 

Precisely.  The one-eighth of an A-level grade finding is statistically significant, but it isn't significant or important in the way we use those terms everyday.  One eighth of an A level grade is not at all educationally significant.  It’s the difference between getting an E and getting a slightly better E. I would think that an eighth of one A-level grade may well be within the margin of error of most examination boards. If you sent off the exam papers of every kid in the cohort to be remarked, you'd probably get a greater fluctuation than one eighth of an A-level grade.  It's the equivalent of 5 marks out of 400 on the uniform mark scheme - that is, 1.25% of an entire A level. And of course, most kids will do three A-levels. So the impact over the entire A-level studies is 0.4%  

Now, there is a question about what the correct average base level is. The IFS express some confusion about this themselves, saying 

'Strictly speaking, this calculation does not compare like with like since the estimated impact is the average effect across pilot areas whereas the base reported is the average level across the whole sample (pilot and control areas). However the choice of an appropriate base is not clear and this calculation provides a reasonable approximation.'

When they break it down by age, sex and race, there is a huge fluctuation in base levels.  I can't definitively find the correct average base level - obviously it isn't as simple as averaging all the ones given in a table because there are more females than males, a lot more whites than ethnic minorities, etc.  So I cannot find the definitive average base level for all pupils, but they do repeat and reuse the figure of 56.7 a couple of times, so that might be it. In which case improvements of 5% are educationally insignificant. Even on some of the higher base levels, where improvements of a few percent mean more, then based on my analysis above we are then perhaps talking about an impact of 1 or 2 percent of the total grades achieved over three years. 

The reason why all these figures are so statistically significant but so real world insignificant is that they start from such a low base. It's like when you have a school of 1000 where one kid gets an A grade. If the next year you can get two kids getting an A grade, then you have a 100% improvement! The headline figure is completely true and looks fantastic, but dig deeper and you see the problem.

If we analyse some of the other 'significant' improvements the IFS talk about, we can see this.  In the news reports on this paper, the impact of EMA on ethnic minorities was trumpeted.  Here's what the impact was:

"by age 19, the EMA increased the average A Level scores of these two groups [black females; Asian females] by around 8.6 points and 16.1 points respectively." I don't quite know how they can guarantee this improvement was in A level scores - I discuss this further below.  That aside, 8.6 points equates to about two fifths of one A-level grade, and 16 points to about four fifths. Be clear, it's not two fifths of one A-level, but two fifths of one A-level grade. It's actually the difference between just getting a D grade, and getting a middle D grade, or just getting a C grade, and getting a middle C grade.  To go back to my uniform mark scheme difference, it's the equivalent of 16 and 32 marks respectively out of the 400 available for one A-level, which works out as a 1.3 and 2.6% impact on a two year three A level course. I think these stats do, just, have some 'real world' educational significance, but still not massively so.  I would also point out that for by far the largest segment of females - white females - there was no statistically significant impact on results. There are similar results for males. So what we are seeing here is - again - that where there EMA does have an impact on attainment, it is concentrated on a very small sector of those receiving it. That is, again, a large amount of EMA is wasted. On balance, I would want to preserve the small but, I think, ‘real-world’ significance of the gains to ethnic minority females. A system that targeted money at those in need rather than giving it to all those beneath a crude income level would get most of the gains of the current system for much less money. Funny enough, that is exactly what the government have proposed.

Two other minor things I would point out as well - a lot of the findings on attainment are less statistically significant than they might be - some only to the 10% level. And secondly, there seems to be quite a bit of uncertainty about the points tariff the researchers talk about. At one point they refer to it as the ''Key Stage Five tariff'. It's a common misconception even amongst educators, but there is actually no such thing as Key Stage Five - at least not in the official sense that there is with Key Stages Three and Four.  Then, at other times, whilst appearing to refer to the same thing, they talk about the 'A-level tariff'. Of course, there is such a thing as A-levels, but I don’t think there is such thing as an exclusive 'A-level tariff'.  I have assumed what they are talking about here is the UCAS tariff, especially as they use the phrases ‘Key Stage Five tariff’ and ’A-level tariff’ interchangeably. The UCAS tariff includes a whole range of different qualifications, academic and vocational. So when I have been referring here to A-levels, don't assume that I am excluding all the kids doing vocational courses. The tariff includes them, and believe me, an increase of 2.5 points is just as unimpressive for BTec nationals as it is for A-levels - arguably even more unimpressive.

Overall, what the findings do here is confirm the anecdotal suspicions of me and of everyone I know who has taught in a school with EMA.  You can split up the kids getting EMA into three groups

• ·         Group 1 - Most of it goes to kids who would have stayed on anyway, and has a limited impact on their decisions about paid work and if it does have an impact on this decision, has a very limited impact on whether they reallocate that time to schoolwork. Hence, it has a limited impact on their eventual grades.
• ·         Group 2 - Of the EMA that does go to kids who would drop out without it, many of them are only there for the money – their eventual qualifications are poor, perhaps non-existent, and unlikely to impact their lives.
• ·         Group 3 - The remainder of pupils on EMA – I’d make an educated guess of about 2-3% of the entire number of kids on EMA, a guess which is informed by although not based on the IFS report – would drop out without it, or would struggle on with reduced grades.  These kids deserve the money. 

 If there were no possible other way to get these kids the money, then I guess you could have an agonised debate about whether it was worth wasting £535-540 million in order to help 12-18,000 kids stay on at school or get better grades.  On the 2% scenario, each kid who has been genuinely helped is costing you 45k a year.  That seems fairly awful value to money for me. I am sure someone will argue that that sum is worth it if it stops those 12,000 ending up on the dole – but then, you’d have to prove those kids would have ended up on the dole anyway, as there is no guarantee they would – I would suspect that the kids who end up on the dole are more likely to be those from group 2, for whom EMA merely delays their entry into NEETdom. Anyway, to prove that you would need to do another complicated survey of the future careers of EMA recipients and find a way of statistically justifying that.  If – and it’s an important if – there were no other way to get these the money, then perhaps, at a pinch you could justify spending half a billion on kids who didn’t need it. I probably wouldn’t buy that argument, but I might concede there was an argument.

Fortunately, there is no need for such an agonised debate. There are other ways of targeting the money at the kids who will make use of it, so the above paragraph is completely academic.  There are several possible ways of targeting money at the kids who really need it, but probably the most effective is to do what the government are planning – give the fund to schools and colleges to distribute in cases of genuine need.  Hilariously, some people have said that this will increase the burden of bureaucracy on schools.  Those people have clearly not seen the burden of bureaucracy the current EMA system brings, which just reinforces my general suspicion that very few of the people defending EMA have ever worked in a school.

I will discuss what this fund should look like in my next post.

Email to Shane Cowen regarding EMA

I have emailed Shane Cowen of the NUS asking him to correct the factual errors in his briefing paper and acknowledge that EMA is spent on lots of pupils who would be in education otherwise. Here is the email - it's based largely on what I say in this post.

I write to you concerning the Educational Maintenance Allowance (EMA). The National Union of Students are campaigning against the planned cuts to this allowance and on your website you have a briefing paper that explains why - http://www.nusconnect.org.uk/pageassets/campaigns/furthereducation/shaneexplains/Shane-Explains_41.pdf

This paper contains many factually inaccurate assertions.  Firstly, you criticise the government for cutting EMA on the basis of a survey by the National Foundation for Educational Research (NFER), a survey you claim is methodologically flawed and one whose findings contradict all the other surveys on the subject.

However, all the surveys you cite do in fact completely back up the findings of the NFER survey. The NfER survey claims that 88% of students would still be in further education without EMA; this is consistent with the findings of the Institute of Fiscal Studies, the CfBT, Ipsos Mori and RCU (links at bottom) and if anything overstates the impact of EMA.  Whilst you cite some of these surveys, you either draw the wrong conclusion from the survey, or, in one case, factually distort the survey data.

When you cite the CfBT study, you claim that it shows that only 36% of EMA spending is wasted. In fact, it shows that at the very least, 35.7% percentage points of EMA spending is wasted. In this case, the 35.7 percentage points equates to 83% - extremely close to the 88% of the NFER research. This error rests on the failure to appreciate the difference between percentages and percentage points – I am happy to discuss this further if you wish. Not only this, but that analysis depends on accepting your point that EMA has increased participation by 7.3 percentage points.  I assume you got this number from the CfBT, who in turn took it from the IFS. What the IFS actually say is that EMA increased participation by 5.5 percentage points for males and 7.3 percentage points for females, which would suggest that actually the overall figure is somewhere in the middle of those two numbers.  Plus, the IFS analysed the participation rates in two different ways, the other of which gave an improved participation rate of nothing for males and 2 percentage points for females. Were I to take those latter figures, it would suggest that only 0% of males and 5% of females would drop out without EMA.

The RCU survey suggests 6% of students would have dropped out without EMA; the Ipsos Mori one suggests a 13-17% figure. The only survey I can find that suggests more than 17% of pupils would be deterred from further education by the cutting of the EMA is one conducted by your organisation. However, on your website I can only find the results of this survey, not its full data and methodology. Given that you have rejected one piece of research because of its methodology, I think it is only fair that you publish the methodology of your survey so perhaps we can see why it contradicts five powerful independent studies conducted by respected research organisations.

The CfBT paper actually defends EMA, but even it has to completely concede the deadweight issue.  Why then do you persist in denying the fact that a large amount of EMA goes to pupils who would remain in further education without it, when even the academic supporters of EMA concede this point? In your briefing paper, you state that it is ‘astonishingly irresponsible’ for the government to cut EMA on the basis of one survey. I think it is astonishingly irresponsible that you ignore or misrepresent four other surveys on this issue all of which directly contradict your assertions.

You do of course go on to discuss the other benefits of EMA beyond increasing participation, and I disagree with you on those too, but for now I would like to remain on this aspect of the issue. On this one particular and important aspect, you have factually misrepresented the impact of EMA. Can I therefore ask that you:

a)      Correct the factual error in your reporting of the CfBT study

b)      Publish the full data and methodology of your own survey into the impact of cutting EMA

c)       Acknowledge that all the substantial research on this issue finds that the large majority of EMA spending is spent on pupils who would remain in further education without it.

I have published this letter on my weblog. Please let me know if you would be happy for your reply to be featured there too. 

Suzanne Moore in the Guardian

Another odd article in the Guardian, this time by Suzanne Moore.

She makes this rather startling statement:

Little has been said about EMA, a means-tested benefit

Little has been said about it? Does Moore read any of the papers she writes for? Type 'educational maintenance allowance' into Google news and see what you get  Look at the Guardian archive on it. And it's not just the left - look at the Telegraph archive on it.  EMA has got its own website and its own campaign day next week. I mean clearly, relatively, less has been said about EMA than tuition fees, but there are probably medium sized pop bands who wouldn't mind EMA style coverage. But let's not let facts get in the way of things, apparently the reason for this shocking absence of coverage is:

possibly because those who live on less than £20,000 a year are not in the middle-class bubble. 

Actually, EMA goes to kids from families on less than £30,000 a year, which is why 42% of kids get it. But in fact, yawn yawn, it's entirely the opposite. The whole reason why such misrepresentation of EMA is allowed to continue is because very few of the people writing about it in the media have ever been inside a school.  They might have met someone who was on EMA, or they might have got a letter from one once, or they might even have been on EMA, in the case of Bridget Phillipson, but they have never dealt with cohort after cohort of kids on EMA. They have therefore never seen the fact that 90% of those kids would still be there without it. They haven't seen the negative effects it has on students, making them see learning as being worthless without cash.  They haven't seen the kids who don't want to be in school who stay on just to get the money and cause lots of people a lot of time and grief. They haven't seen the bureaucracy associated with it.  And because they don't really know anything about how it works, they haven't bothered to look closely at the research evidence which backs most of this up. So these people are then capable of making such astounding statements as:

To remove this in effect prohibits a whole sector of society even getting the qualifications they need to get university.

EMA has only existed this decade.  We all know that before it existed, poor kids still stayed on in the sixth form.  I guess it is understandable, if still frustrating, that 20 somethings make statements like this, lacking in all historical perspective. But Suzanne Moore was a sentient adult before 2000. She probably personally knows people who went through sixth form before EMA existed - or maybe she doesn't, and maybe she's the one in the middle class bubble. In which case - I am here to tell you, from my own experience and from all the stats I have read on this - poor people stayed on at school before EMA existed. Fact.

And look at the use of that word 'prohibits' - how absurd. To prohibit something is to pass a law that bans it.  No one is 'prohibiting' a 'whole sector of society' from further ed.  In fact, there's a reasonable case for arguing that the main thing that stops kids carrying on in further ed isn't lack of money, but lack of prior educational achievement, which is why EMA has barely dented the NEET rate. In which case, redirecting EMA to the Pupil Premium will do a lot more to get that 'whole sector of society' into college.

But as if all that wasn't enough, we then get this.

The students at least know their figures

No, no they don't - and neither does Suzanne Moore.

What are they going to do when the world doesn't fall in?

Here is Toynbee again, engaging in what I would guess is her second favourite activity after picketing Top Shop - spreading misinformation about EMA. I've had her up for this before, when she called the cut 'wicked' and you can see more hyperbole from her here and here. This video is the best though - she manages to call it pernicious, a tragedy AND a travesty. Wow! Tragedy and a travesty? Someone tell Sophocles, Toynbee's just reinvented literature.

When writing a lot of these articles about EMA, I am astonished at how easy the claims of the NUS, Toynbee, etc., are to refute. I don't mean that they're easy to refute in that I have made them change their minds, of course, but I mean how easy they are to refute in that they completely oversell their case.  If someone came to me and said "look, I know that there's abuses of EMA, I know there's problems, but I really think it is worth having a system where you pay a bit - even a bit over the odds - to get some of these hard to reach kids into further ed" then you could have a reasonable discussion with them and you would probably end up with a system close to what the coalition have proposed - a discretionary fund allocated by need, not by a crude means test.

But no one defending EMA makes that reasonable case.  Instead, they claim that abolishing EMA will lead to the world falling in. It is evil, wicked, devastating, punitive, cruel, unfair, pernicious, a tragedy and a travesty. That's what makes their case so easy to argue against.  I can very simply show that it is none of the above. Actually, if they made a more limited case like the one I put forward above, it would be a lot stronger and probably end up doing a lot more for the poor kids they claim to represent - but that wouldn't get them any newsprint.

The other thing about this invective inflation (apart of course from such amusingly bathetic lines as "these cuts are punitive, cruel, unfair and inadequately evidenced" and "these cuts are a tragedy and a travesty") is that it means they won't have any words left when the coalition actually does something bad. They're crying wolf. Blimey, if the coalition invade Iran and Pakistan and get us bogged down in two quagmires of a war, there won't be any language left to use for that.

I understand that a lot of these NUS hacks are young men in a hurry and have careers to make, but don't they ever stop and think tactically about what they're doing? What are they going to do when further ed participation continues on its gentle upward slope? What are they going to do when higher ed participation stays constant or increases - as it did after the introduction of tuition fees in 1998? And it's not just in education. In so many areas the Labour left are making predictions that they must realise hold them massive hostages to fortune. It's all very well chucking wild hyperbole about, but when you actually start making predictions, then you're letting yourself in for problems. A saner member of the Labour party made this point -

'In fact we’ve stopped acting like an opposition, and taken up a new line in economic Tarot reading. “The government’s plans will push us into a double dip recession” – Ed Balls. “The Government’s announcement will make a million people unemployed’ – Angela Eagle.

And what if Osborne doesn’t turn over the Grim Reaper? The economy staggers on with marginal growth. Unemployment ‘only’ goes up by half that figure. Our bold predictions will make the axe man look like a skilled surgeon.

We’ve rolled the dice. Instead of neutralising the reduction issue by aligning ourselves with the net Tory deficit level and timetable, we’ve bet the house on fiscal meltdown...What should have been the greatest gamble of David Cameron’s political career has become Ed Miliband’s.'

If you go around claiming that we're in for a double dip recession, a rerun of the Great Depression and mass social breakdown, then you make a jobless or low-growth recovery look like a success for the government.  If you go around claiming that abolishing EMA will destroy the life chances of half a million kids, then you make an unchanged NEET rate look like a government triumph.

You also alienate lots of people like me. I am a Lib Dem, but in an AV situation - which we already have in London, where I'll be voting in 2012, and which we might have nationally - Labour need the votes of people like me.  I used to be a Labour supporter, I am a public sector worker and I am from a working-class background. I want to support Labour.  I admire a lot of their traditions. The fact that if things carry on as they are I will vote Tory for my second preference is not just because of the coalition but because Labour have lost touch with reality. I work in the public sector, I care about the kids I teach, and I resent being told I am 'evil'  because I support an evidence-based policy decision that has pretty much the same aim as the previous policy.

More EMA contradictions - do its defenders even read the evidence they link to?

So I showed here in great detail how Shane Chowen, the NUS vice-president, has managed to cite two academic papers that he claims back up his points about EMA when in fact they completely undermine them.

Here we have another example of this. James Mills, founder of the Save EMA campaign, and Ed Miliband's new best mate, starts off in this article by citing the NUS survey that showed 65% of participants couldn't carry on studying without EMA. Then, eager to convince us yet further, he says:

In addition, RCU Market Research Services carried out research on the national scheme and published a report called Evaluation of the EMA National Roll-out 2007, which concluded: "EMA is reducing Neet (those Not in Employment Education or Training) and also motivating learners to work harder." Ipsos Mori published a report in 2008 which reached similar conclusions.

Really? Let's click on the links and see. The RCU paper actually says '6% of respondents said they would not have continued in learning without EMA.' The Ipsos Mori paper actually says '13-17 per cent of respondents would not be in learning without EMA'. That is, between 83% and 94% of EMA spending is wasted - this correlates with all the other evidence we have which I outlined here.

So, papers cited by the head of the Save EMA campaign directly contradict one of his central points. Has he even read them?

NUS on EMA - lies, damned lies, statistics.

One of the biggest claims against the EMA is that of ‘deadweight’. That is, whilst EMA might persuade some kids who wouldn’t have otherwise stayed on to do so, an awful lot of kids who get it would have stayed on without it. Therefore, a large chunk of it is wasted. 

Anecdotally, this completely chimes with my experience. I would guess about 90% of the kids I have taught who received EMA would have stayed on without it.  Some research does bear me out – in a rigorous survey commissioned by the last government and published by this one, the National Foundation for Educational Research suggest that 88% of pupils receiving EMA would stay on without it. 

On their website and in the Guardian, the NUS reject that survey for methodological reasons, some of which I believe do have some foundation. They also cite their own research which shows only 45% of pupils would be able to continue without it.  However, I cannot find the methodology or the provenance of this research. All I can find is a page on their website that suggests all they did was set up an online survey and ask students to respond.  I hardly think the way to respond to a survey you think has methodological flaws is to conduct one with many more methodological flaws. I am therefore going to ignore this research.

 They do however cite two respected academic papers as backing up their point, one from the IFS and one from the CfBT education trust. Based on these, they make this this semi-literate point attacking the decision to axe EMA:

To make such a far-reaching decision based on one survey, particularly when all other research  - including that by the Institute for Fiscal Studies, which found EMA increased participation by up to 7.3 per cent - is astonishingly irresponsible.

Let's have a look at this other research then. The IFS did indeed find that EMA increased participation by up to 7.3% percentage points. That 'up to' is quite misleading, as from what I can see the IFS say that EMA increased particpation by 5.5 percentage points for males and 7.3 percentage points for females, which would suggest that actually the overall figure is somewhere in the middle of those two numbers.  Plus, the IFS analysed the participation rates in two different ways, the other of which gave an improved participation rate of nothing for males and 2 percentage points for females. Anyway, I'll take the most generous 7.3% because I am generous, and even if you do this, then far from contradicting the NFER, this almost completely reinforces its point about deadweight.  43% of all 16-17 year old students get EMA. Reduce that by 7.3 and it leaves 35.7 percentage points of kids who are getting EMA but who would have presumably still been at school. Note I have italicised percentage points. Little stats lesson here for the NUS – 35.7 percentage points is NOT the same as 35.7%.  Allow me to explain with an example:

We have 100 sixth form students.  35.7 get EMA but would be at sixth form anyway. 7.3 get EMA and wouldn’t be in the sixth form without it.  The other 57 are at sixth form and don’t get the cash. Of the 43 kids who get EMA, it’s only 7.3 kids who are actually being incentivised by it to stay on at 6^th form. The other 35.7 would be there anyway. Thus, the deadweight cost is 35.7/43, or 83 % - almost exactly the same as my anecdotal guess and the more rigorous NFER survey! That is, the NUS cite a paper and a statistic that entirely supports the theory they are trying to knock down! And that's with me selecting the data that most supports their case - if you take the 5.5 percentage point figure that the IFS say is the male improvement rate, or if you take their figures from the other type of analysis they did, then the deadweight problem looks even worse.

As if that were not hilarious enough, the NUS also quote from another rigorous survey of EMA which defends its deadweight costs. I disagree with this paper, but it is certainly numerically correct. It defends EMA on the deadweight charge not by pretending deadweight doesn’t exist, but by saying that there is a lot of deadweight in a lot of other policy areas. This appears to me a very weak argument, but I will not get into that here. The point is that again, a paper the NUS cites as refuting the deadweight charge actually agrees that there is significant deadweight. Even more amusingly, the NUS manage to completely mis-cite it – either deliberately or because they are innumerate.  The paper actually says this:

Unlike most other areas of policy however, the level of deadweight in EMA allowances can be calculated (which may be the reason why it is the focus of attention). The most recent figures show that 43% of full-time students aged 17-18 receive EMAs. Taking an average of recent evaluations EMA may have increased participation by up to 7 percentage points, leaving some 36 percentage points as deadweight. Although probably better than many other areas of policy this may still be though too high.

You’ll notice that this says 36 percentage points – as I have shown above, in this case, those 36 percentage points equate to 83 percent. But how is this statistic reported in the NUS briefing paper? Yep, you guessed it:

the work carried out by Mick Fletcher, a leading authority on EMAs, where he found that the 'deadweight cost' is about 36 per cent.

What are the chances of kids who can’t spell incentive being able to work this out? Remote, I would say.

I wonder if the NUS are being ‘astonishingly irresponsible’, or simply innumerate.

I despair, I really do.

Out of touch protestors...and Guardian, Red Ed, Lisa Nandy, CiF, etc.

So the Guardian claim that the student protests yesterday were justified, Red Ed wants to go and join in and Labour MPs are queuing up to show their support.. If you look at the comments on CIF, it would suggest that there's a lot of public support for the protests. I am not so sure. I've already blogged about how small these protests are as a percentage of the UK electorate. In the staffroom of the school where I work - which is by no means a right wing hotbed - there is a great deal of bafflement at the protests and protestors.  I think the first one had some sympathy, but the others have just seemed like a chance for kids to skive off and muck about in London.

Added to that, a lot of the people I work with - and a lot of normal people in general - didn't have EMA, did have to pay some element of fees, did have to work or bring up children whilst they went to university, and certainly haven't had anything like the cushy lifestyle these students seem to have.  Most people look at the protesting students and see people who are more privileged than them, not less. And they're right. I keep harping on about it, but it's a complete myth to claim that the "older generation" had their fees paid for them.  About 5-10% of the older generation had their university fees paid for them.  The other 90-95% didn't go to university. Even as university places have expanded, that's come at the price of people paying fees, doing part-time degrees, going back into study whilst having a family and holding down a part-time job.