I’ve argued below that the housing bubble was like an economy-wide Ponzi scheme. But how did it start and why was it allowed to keep inflating? I think the reasons are mostly political.
For the bubble to even start, you had to have major changes to mortgage regulation, which allowed people to get huge loans, leading to an "arms race" effect which had a key role in bringing about the bubble. This happened mostly in the UK, US and a handful of countries heavily influenced by these two (Ireland, Australia, Spain). These changes were in turn an important part of the conservative shift in these countries. Reagan, Thatcher, Aznar, Howard all made a big deal out of the notion of an "ownership society”. Politically this had the effect of making people on quite modest incomes feel some kinship with the rich: after all, if you own your home and maybe even some stock, aren't you one of “them”? (Among other things, this led to counter-intuitive effects such as poor people seeming to support policies that hurt them, like the repeal of inheritance tax, and reductions in capital gains.) I agree it's hard to make a causal argument here, but the political and economic changes were clearly coupled at some level.
But let’s be nice and assume that the origins of the housing bubble were random and unplanned. I would still argue that the decision to keep it inflated was largely political.
The crucial thing was the collapse of earnings growth for middle-income people. Remarkably, in recent years, in the US in particular, real median income didn’t grow at all even as productivity grew, mainly because the top were taking such a huge slice (as beautifully documented by Piketty and Saez). Harvard's Elizabeth Warren has shown very carefully what this meant at the household level: budgets became much, much tighter, with disposable income (after basics) going down: money tight in (what should have been) a time of plenty.
If you're in government the right thing to do would have been to start to address the reasons for stagnant incomes (i.e. look at executive compensation, the City, tax loopholes etc.). But if, for ideological reasons, you've already decided you don't want to talk about those things, that route is closed to you. Yet if people are really struggling, on a day to day basis, you’ll lose the elections. So when people start using their homes as ATMs and so on, you encourage it. This allows you to achieve something extraordinary: to engage in a massive transfer of national wealth to the rich and win popular elections at the same time*.
*Unfortunately, the downside is net destruction of national wealth, but by then you're out of office, or you pretend the whole thing is an exogenous shock, like a hurricane.