Further response to Anthony Painter

I am in italics.

Thank you for the response. And you are right to say that my major issue is with your analysis rather than your statistics.

Yes, ultimately I agree that our disagreement is analytical, not statistical, but some of the things you say below, whilst not actually impacting that much on either, are still such misrepresentations that I have had to comment about them.

The problem I have with your statistical ‘dancing around’ is that in previous posts you have flipped around between pilot area averages and EMA recipient averages without making it clear what you were quoting and why. That weakens your case considerably.

Does it? Does it really? I assumed my readers had some intelligence – which you clearly do – and that weakens my case? If there was a discrepancy of several times in the stats I was quoting, then you would have a point, but in my follow up post I do an average of all the EMA recipient averages and it comes out at 5%, which is even less than my original post claimed. The reason why I jumped around a bit was actually because I was following the lead of a lot of EMA defenders, such as those in the NUS and CfBT, who seem to have dived into the paper and picked the biggest percentage they could find. Because I was dealing with their analyses, I obviously needed to use the figures they’d picked so I could follow through on their analyses. So when I jumped around, yes, actually, I was ‘weakening my case’ in a different sense to the one you mean because I was deliberately letting my opponents pick the figures that best backed up their argument. Show me one place where my jumping around resulted in a significant discrepancy that favoured me?

The problem I think you are encountering is that you are trying to two things at the same time and, as a consequence, you end up achieving neither. The first thing you are trying to do is administer a statistics lesson (to the NUS presumably.) But your posts have demonstrated how easy it is- through ill-discipline-

!!!

to give a distorted view.

Right, just because I jumped around DOES NOT mean I distorted. Again, show me one area where my jumping around significantly distorted the case? The only places I can find where it did are, as I state above, where I distort in favour of EMA in order to track the analyses of pro-EMA reports.

Your clear statistical capability is masked by unclear presentation which devalues

How does it devalue?

that element of your case. (and thanks for the patronising aside on Appendix D! I did as it happens read Appendix D though Table 1 is pretty self-explanatory anyway and the accompanying text that I quoted makes it equally clear. It should be noted that Table 1 (as opposed to 1a and 1b) deals with participation only and not attainment.)

OK fair enough, that may have sounded a little patronising. But if you had read appendix D, why did you say in your original post:

The difference between the area and individual impact of EMA.

The figures you mainly quote above are pilot area averages (thou you do jump about a bit). It is important to read pp 6-7 of the IFS report to understand why this is problematic. The key point is:

"We therefore suggest as a rule of thumb that, in order to obtain the effect of the EMA across those who received it, the estimates above be multiplied by a factor of 2½ (for outcomes at age 16) or 3 (for participation at age 17 and attainment outcomes)."

So the participation and attainment impacts are in the main much greater than you suggest.

That bit above in bold is the only thing you said in your original post about area/individual impacts, and you do say:

 So the participation and attainment impacts are in the main much greater than you suggest.

When in fact as I showed in my first response and as you would have known had you looked closely at table one and appendix d that therefore the participation impacts WERE NOT much greater than I suggested and DID NOT have to be factored up.

In particular, you criticise the NUS (rightly in fairness)

I am still waiting for their response. And this is not a trivial matter – as much as you think EMA matters even on the basis of the facts you cite, the truth is that there is a massive difference between what you as a numerate person are claiming about EMA and what the NUS are claiming. Now, you are still backing EMA despite the fact that you realise it is much less effective than the NUS claim, because you still think it’s effective enough. But I think if everyone were clear about how effective you think it is statistically, then there would be a lot fewer people willing to back it.

 on their confusion of percentages v percentage rates. But yet you flip between the two over the posts here without making it clear what you are doing and why. In your case, it’s not confusion; it’s lack of clarity.

Final point on the analysis, the point you make about the impact of EMA on different ethnic groups is, well, ill-considered. A greater proportion of certain groups receive it and so by looking at these sub-groups we can get a better sense of its real impact (though there may be issues specific to certain groups, it is highly likely their socio-economic status is the major impact on likelihood to participate in education.) So yes a majority of people who receive EMA are white British but a majority (probably) of, say, black African British receive it. My point was quite simply that if you want to understand the real impact of the policy you’d be better to look at the impacts on the latter than the former group. So it’s nothing to do with racial targeting etc. Quite why you went down that route is difficult to fathom.

Not sure I get what you mean here – as I’ve established, for participation I am only looking at EMA recipients, not the entire black or entire white communities. This analysis seems to show that it does have a bigger impact on ethnic minorities who receive it as opposed to white students who receive it.

Part 2:

Anyway, the major issue with the analysis is not actually to do with statistics- though it should be emphasised the issues presented by your use of statistics of are not insignificant. It is your reliance on the ‘deadweight’ argument. 

All public programmes have a degree of ‘deadweight.’

Yeah, you’re right. A degree! Not 90 percent! If that’s a ‘degree‘ I’d hate to see what you thought a ‘large degree’ was.

The example I gave on my piece on Left Foot Forward was on GP check ups. Only a tiny proportion lead to the identification of a serious illness so the vast majority is deadweight so would you cut this expenditure? 

This is a terrible comparison that tells us barely anything about EMA.  Firstly, just because only a tiny proportion of GP check ups lead to the identification of a serious illness, does not in any way mean the rest are wasted!  Frequent GP check ups have a preventative function- surely you realise this? Secondly, even if you are sceptical about how easy it will be to identify the kids who don’t need EMA (I deal with this point below) surely you’ve got to realise it’s a damn sight easier to work out kids who don’t need EMA than it is to work out a stomach pain that’s cancer or a stomach pain that’s a dodgy bit of meat? Even the CBfT paper which supports the EMA fully concedes that you could abolish the two lower EMA bands without significantly affecting the drop out attainment rate – can you at least accept that? That there are very easy and simple cuts you can make that won’t lead to anyone dying of cancer?

You could say the same about education- more than half don’t leave school with 5 GCSEs A-C (including mathematics and English) so why not just not bother with the rest and put them on some cheap literacy and numeracy programme with some basic subject matter, teach them two hours a day, and reduce costs by over 50%? There is nothing intrinsic in your statistical analysis to suggest there is too much ‘deadweight’ in the programme.

Another dreadful comparison. I happen to believe every kid is capable of getting 5 A*-Cs. Even the ones who don’t – I believe in my naivety that they get some value out of doing GCSEs anyway. The money we spend on them is not deadweight. The money we spend on GP check ups that don’t lead to identifications of cancer is not deadweight. I don’t think you quite get what the definition of deadweight is. Literally, ‘Expenditure to promote a desired activity that would in fact have occurred without the expenditure.’

I’ll give you some other examples of programmes that have deadweight in, and I think there is a good case for cutting a lot of them too. Winter fuel for pensioners is probably the best example. Here, the desired benefit is that all pensioners should be warm in winter, when in fact lots of them would without the cash. Likewise a lot of the pensioner benefits fall into this category – TV licences, travel. Child benefit too – although here it is less clear because there are differing claims about what the actual desired outcome of child benefit is. If it was to prevent child poverty or give all kids a little treat now and again, then there is clearly a lot of deadweight, but if the aim is something more grand – about saying that society shares the responsibility of bringing up children, then actually there probably isn’t any deadweight. Deadweight by its very definition depends on what your desired outcome is – in the case of EMA it is nice and easy because the govt told us from scratch it was participation – although I they widened that, rightly I think, to include attainment as well. Based on this proper definition of deadweight, can you think of any social programmes that have a greater degree of deadweight than EMA? Or even the same?

It’s just a number and you’ve arbitrarily said ‘that’s too much.’

Well yes, in the sense that 18 is an arbitrary cut off to vote, 17 is arbitrary to drive. You’ve got to have cut offs somewhere.  Personally I think you should seek to reduce any element of deadweight and then if you can’t reduce it below a certain point – I’d go for about a half, which you’re right is arbitrary – then you have to consider the value of the programme itself. Some programmes will have high deadweight that can’t be reduced that are still worthwhile. I cannot think of any at the moment, but I am prepared to believe there are some. EMA isn’t one of them.

But the logic of your position is the culling of many critical programmes that I’m sure do work in your view.

Such as? Give me one example of a genuine case of +50% deadweight, and chances are I think you could either make savings or cut. There might be one or two exceptions, or you might come up with your dreadful examples of GP check ups again, but overall, I think I’d cut.

For some reason, you’ve jus decided to single out EMA (perhaps because it’s a programme that the coalition wants to cut?)

I think every social program has to have its costs and benefits appraised. I think there is definitely a case for cutting some of the programmes I mentioned above, but I would agree with Neil O’Brien that EMA is one of the least effective ones out there. The reason I have made such a fuss about it is partly because I was absolutely baffled by the passionate defences it has received, but also because, in a very human and one-eyed sense, it affects me. You’re right, there are other programmes that are as bad  (although not nearly as many as you seem to think) Giving winter fuel to Fred Goodwin is just as outrageous, but I don’t have to deal with the consequences of that every weekday. I expect if Fred Goodwin and Polly Toynbee sat at my work every day whinging about losing their winter fuel, I’d start to get pretty aggravated about that too. The reason why I am so het up about EMA is because, as a teacher, it affects me. I am amazed by the fact that every teacher I know, regardless of their political leanings, thinks EMA is a bit of a racket, and yet the defence of it in the media has been so impassioned. I posted something like this on CiF recently and one respondent went out of her way to say how wrong and unfair and evil she thought the Lib Dems were, and she wouldn’t want anyone to think she liked me or supported me, but on this particular issue...she thought I was right.  Unsurprisingly she was a teacher.

PART 3

My final point is an observation. Throughout your analysis you have taken a very school-centric view. This is problematic in a number of ways. Though I haven’t got access to the numbers, given the nature of the cohort we are talking about, a larger proportion will actually be in college rather than school. This creates issues for your argument in a number of ways. 

Firstly, College qualifications are measured by percentage achievement at various levels rather than UCAS tariff points.

You’re right I haven’t taught in a college, but what do you mean by ‘college qualifications’? Do you mean NVQs? They are the only significant qualification offered in a college that aren’t included on the UCAS tariff - and in fact the NVQ in accounting is. The last time I looked the bulk of college qualifications were included on the UCAS tariff, but I am happy for you to tell me otherwise.. That is the whole point of the tariff – to provide equivalences. If you look here, you’ll see how many qualifications the tariff includes.  For all of them, the average 2.55 points improvement EMA brings is tiny, and even more tiny for vocational courses than academic ones in a lot of cases.

If say, an intervention increases success by 7% as seems plausible from the IFS analysis, this is enormous. A college that increases its achievement rates by that amount could go from being in the bottom to top quartile! (Incidentally, had the IFS used success rates rather than achievement rates, drop outs could have been factored in: the rate is calculated by achievement x retention. This doesn’t mean that scaling up of achievements is a problem that can’t be surmounted which you to seem to suggest- in fact, p.7 of the IFS report states explicitly that they can be, albeit in a rough and ready fashion.) I would encourage looking at the by college evidence presented in this article in TES: http://www.tes.co.uk/article.aspx?storycode=6064883

This is another classic sleight of hand article - most of the figures it cites are about kids staying on and ‘completing’, which is a lovely euphemism. It does state that there was a better pass rate, but how much better and what the average pass rate was isn’t mentioned, and nor is the real world use of the qualification – ie, if it has led to them getting away from NEETdom or just postponing it.

Secondly, your assertion- and that is what it is- that headmasters can allocate the fund efficiently may be so (though I some scepticism- it opens a scope for gaming of the system where students claim that they would not continue/ drop out unless they receive the payment but the EMA suffers from gaming also in fairness.) However, move into a College scenario where you may have 2000 students (Lambeth has 2,078.)

This is all you need to know about Labour’s love of centralisation. A headteacher in charge of 2000 kids won’t be able to do a better job of allocating resources than a central bureaucracy in charge of 600,000? Yeah, all right, that’s an ‘assertion’, of course there is no evidence for it.

I’m afraid your optimism re efficient allocation becomes impossible in such a scenario (Lambeth College will have 1000 students on EMA!) I should state that in the LFF piece I stated clearly that £50million was better than nothing and £100million would be even better still! 

What all this means in practice is that this policy is not a matter of ‘spending £500million to get £50million benefit.’ The only thing to back this up is your ‘deadweight’ argument which, as I have shown, is a nonsense.

No you haven’t

Actually, the overall benefit is significantly greater than that. The cost to the taxpayer of a NEET is £56,000 over their lifetime. So if just 9,000 end up as NEETs as a result (hands up, I doubled the expenditure over two years in the LFF piece but should have also divided it by two realistically given it’s a two year cohort which meant I quoted 18,000 instead of a more realistic 9,000- silly me to hinder my case in that way!) of the cut then the policy would pay for itself in fiscal terms (and we haven’t even begun to discuss the individual and social benefits….) Given a youth unemployment rate of somewhere in the region of 20% it will be more than 9,000 who become NEET.

Obviously all this assumes that kids getting EMA won’t become NEETs when they have finished. It may sound depressing but the facts and my experience suggest that for a lot of kids, EMA merely postpones NEETdom rather than abolishing it.

So if your argument is that we don’t want to pay now to save later and are willing to take the social costs etc. on the chin over and above that then fine. By why all this statistical dancing to get to that point? I say invest now, save later, unleash individual opportunity, and minimise negative social impacts of low or no qualifications and unemployment. Or don’t. That’s the choice.

You forget as well the problem of opportunity costs. Suppose we could say that EMA did have a minimal and overall positive impact. If we could spend that money anywhere else in education and get a greater return, then EMA is still a failure. One of the other things the IFS report shows is that the reason EMA didn’t have an impact on a lot of NEETs was because their real barrier to achievement wasn’t money, but prior achievement. So therefore a programme which put money into improving attainment at 11-16 could potentially have a better return on investment than EMA. I discuss the problem of returns on investment further here.

And one final thing – the reason why I’ve always felt EMA needed an exceptionally high success rate was because of the significant moral qualms I have about the concept. What’s your view on these?


UPDATE: Just want to say that I do appreciate you taking the time to post on this and at no point have I intended to be rude or patronising, so apologies if it comes across like that.