I was sent a link to this article yesterday by a friend (who works as a fundraising consultant). She asked me what I thought about it ‘from a data perspective’. Given that the sample was so small, did I think it was indicative of anything useful to know? Its headline was:
‘More than a fifth of people plan to reduce regular donations.’
Now, the research says it questioned ‘a 496-strong nationally representative sample in June and July’. All sounds good so far. It’s not a massive sample, but if you think about it – asking 496 people requires quite a bit of time and effort. Compare that to the 5,100 households that represent the nations TV ratings and it doesn’t look too puny either. A great deal of money, careers (and egos) are dependent on that sample size.
It isn’t the sample size that strikes me as a bit odd. It’s the ’nationally representative’ bit that I find misleading. Don’t get me wrong, it is obvious why a researcher would do that. This isn’t a contentious point; it isn’t wrong, but I think the conclusions are a bit misleading because of it.
To draw conclusions about something general from a general group makes sense – ask a nationally representative group about if they feel happy or sad will be a good indication about the general ennui or euphoria of the nation.
But drawing general conclusions from something rather more discreet isn’t such a useful thing to do. If you were polling people about who they were going to vote for, would you want to know if they were legally able to vote, and if they were, would they vote? If you didn’t establish these groups and incorporate that into your analysis, you will get something like misrepresentation. And of course, you get the problem of people not telling you truth, for one reason or another.
So to ask a nationally representative group about something that is, essentially, not evenly represented by the nation (not everybody gives) and even then, not to the same extent (even in the same socio-economic group, not everybody gives the same), I think, is a bit misleading. Donors are neither ubiquitous nor equal (in terms of sheer capacity to donate, that is). And I think of giving to charity as very similar to being cool – anyone that bangs on about how cool they are, probably isn’t very cool.
If you took your 496 people in this sample and worked out how much they gave to charity to start with, I could almost guarantee that only a small proportion gave ‘a lot’ of money to charity. I would say that about 20 percent of those people would fall into the ‘giving a lot’ category and would represent about 80 percent of all the money given to charity by the group as a whole. That’s the general Pareto principle – and it is usually pretty much accurate for most charities when you analyse what proportion of their income comes from what proportion of their whole donor base. Not all charities, I would add, but most.
What do the top 20 percent of people say they are going to do? Telling me that 23.1 percent of people would ‘give a lot less’ would be extremely worrying if that included all your top 20 percent plus a few more. If it was the bottom 23.1 percent, then I am less worried, to the point where I am not worried at all. The short term will be affected, ever so slightly, of course, but it will return, along with the economy, in the mid to long term. We need a bit of clarity in terms of what these people are doing. Suppose the headline read:
‘More than a fifth of people who don’t really give that much plan to give a bit less in the short term but it will all be good in the mid to long term, so be aware of it, but don’t worry too much.’
Obviously the sub editor would have to be fired, but you would sleep a bit easier wouldn’t you?
All I am asking is that if you are going to publish some research that focuses on a rather bleakish statistic, at least let me know which fifth we are talking about.
As someone rather succinctly put it to me once when I was wondering how much of my income I should put into a pension – it doesn’t matter Adam, four fifths of **** all is **** all.
