The Government this week released a report evaluating the Help-to-Buy Equity Loan Scheme. It said that the best estimate showed 43% of the new homes funded by the H2B Equity Loan Scheme were additional and this additional boost amounts to 14% of total new-build output.

That surprised a lot of people.

So I thought I’d better have a look at how this “best estimate” was generated.

I’ll say now it left me with a few questions and not a little concern.

So how did the researchers come get to 43%? The brief answer is that they quizzed people who bought homes through the H2B scheme and derived the figures from the answers.

The report’s objective seemed to be estimating the level of additionality produced by the policy intervention.

Interestingly, at the outset, they make the sensible point: “There are inherent challenges in making an assessment of additionality, the introduction of the policy in April 2013 means it is not possible to establish any meaningful counterfactual and disentangling the effects of the policy from other related policy initiatives add further complication.”

Then they go on to seek a best estimate. I’ll not try to unravel the whole process. I’ll just look at a little bit, but an important bit.

They selected three questions taken from their questionnaire to derive the 43% figure. These were:

Q18a – I would have been able to buy a property I wanted anyway without this assistance.

Q19a – do you think you would have been able to buy this same property without the assistance of the Help to Buy Equity Loan scheme or not?

Q19b – and do you think you would have been able to buy a similar property that was NOT new build and being sold by its owner. By similar I mean in terms of type, size and location?

For question 18a the interviewees could answer: Strongly agree; Tend to agree; Neither agree nor disagree; Tend to disagree; Strongly disagree; or Don’t know.

For question 19a and 19b the interviewees could answer: Yes – Definitely; Yes – Probably; No – Probably not; No – Definitely not; or Don’t know.

For “additionality” the respondents had to not agree with Q18a and not answer yes to Q19a or Q19b.

At first sight that might seem ok, if you trust the answers given.

That is until you look at the footnotes. You notice that the “disagree” portion from Q18a appears to include everyone who said “neither agree nor disagree”, “tend to disagree”, “strongly disagree” and “don’t know”. And the “no” portion for Q19a and Q19b includes “don’t know” responses. Unless, of course, I’m misreading the footnotes.

Without the fine data it’s hard to know exactly how the number giving negative responses to all three questions (215 out of the total 501 sample) would have shrunk if we took out the “don’t know “or “neither agree nor disagree” responses.

The two points to note here are: firstly; that the more negatives there are the bigger the additionality; and, secondly, surely you should exclude “don’t know” and “neither agree nor disagree” responses in these results?

I’m puzzled as to why they would be included. But what unsettles me more is the lack of critical examination of the likely behaviour of the respondents. How would you expect people to respond to these questions?

Firstly, all those interviewed were aware that they were taking part in a survey about the H2B Equity Loan Scheme and, having benefited from it, we might reasonably assume they view it favourably and will, in a sense, feel indebted to it. This will surely shade their responses.

Cognitive bias will be at play, if only because we are asking people to compare what actually happened with a counterfactual that is for many, if not all, unknowable or unexamined.

Think how many times people have said to you: “I couldn’t have done it without your help.” Think how many time you have known full well they would have found an alternative way, if push came to shove.

For the respondents H2B was part of what got them the home they wanted. It is hard to know whether they would or could have taken a different route and bought the home they ended up with (for example, the “Bank of Mum and Dad”) or bought a smaller place or bought somewhere else, if the scheme wasn’t available. They did not need to look elsewhere for alternatives, as those before H2B might have done, so the likelihood is that alternative options were not examined, explored or experienced.

In these circumstances what does “I tend to disagree” or “probably no”, let alone “neither agree nor disagree” really mean?

Personally, I’d be uncomfortable about taking any response less categorical than “strongly disagree” or “definitely no” as a sufficiently strong negative response to the question of whether they would have bought a new home without H2B. And I’m purposely avoiding getting into speculation about what incentives house builders would have provided in the absence of H2B. There would have been other incentives.

So, if we were to take out the “probably no”, “tend to disagree”, “neither agree nor disagree” and “don’t know” responses, what would be left? Far less than 215 out of 501 (43%) I wager.

I sense you could be getting bored here. So I’ll cut it short. I could go on. For instance, I might suggest the use of Land Registry data for factoring the proportion of new-build homes bought with H2B is questionable and makes the additionality provided by H2B look more impressive than if we used potentially more realistic alternative data.

I’m not suggesting that the researchers deliberately set out to stretch the numbers and there may be something in the analysis that I’m missing, but I feel far from comfortable with the best estimate provided.

So when you hear in future the 43% of homes bought with H2B Equity Loan Scheme were additional and that this boosted new-home output by 14%, it may be worth being a little sceptical. Well, a lot sceptical.