Big data and the emperor's new clothes

“London is too full of fogs and … serious people …Whether the fogs produce the serious people, or whether the serious people produce the fogs, I don’t know.”

That was one of Oscar Wilde's bon mots in Lady Windermere's Fan.  It could have been used, and perhaps should have been, in the Observer yesterday in a tremendously important article about big data and its threat to democracy.

Because the real question, as it was for Wilde, is what causes what?

The article was by Evgeny Morozov and I hope everyone reads it.  It isn't so much about the threat from surveillance, important as that is (we should all own our own data, it seems to me), but about the great utilitarian dream that goes along with big data - that somehow all the issues of government can simply be measured, without debate or awkward democracy.  This is what Morozov calls 'alogarithmic regulation'.

The trouble is that he doesn't really go far enough.  He assumes this is somehow a new phenomenon, whereas it actually dates a long way back.

As I wrote in my book The Tyranny of Numbers in 2001, and as Morozov says, all these screeds of data never take you back to causes.  They will never actually allow you to debate them, or to tackle them, just an obsession with symptoms.

It is a New Labour fantasy of handing government over to machines that measure data - but with no understanding of how inaccurate data will always be if it is chained to definitions, or if it leads to controls.

Why does data have such an appeal to modern governments? Partly because of the technocratic thrill of measuring the ebb and flow of symptoms as if government was a gigantic, though not particularly well-oiled machine. Yet cause and effect is the one thing it is quite impossible to measure - interpreting the burgeoning wealth of data to work out what causes what is always a matter of judgement, common sense and intuition.

That's the problem with data.  It won't interpret. It won't inspire and it won't tell you what causes what. Statistics have nothing to do with causation, the pioneering number-cruncher William Farr told Florence Nightingale in 1861: "You complain that your report would be dry. The dryer the better. Statistics should be the dryest of all reading." 

 But over-reliance on numbers sweeps away your intuition along with ideology. It leaves policy-makers staring at screeds of figures, completely flummoxed by them, unable to use their common sense to interpret the babble of competing causes and effects – unable to tell one from the other.

If men with long ring fingers are subject to depression - as they are for some reason - that might alert you to looking for a causal link. The same is true of other peculiar numerical links: high stress makes you much more likely to catch colds, accident rates among children double when their mothers are miserable. 

 These odd connections might surprise and inspire you to think about problems in new ways, but it won't tell you what causes what. You will have to use your intuition to work out where to look in a massively complex world of complex systems. "Scientists try to avoid emotions and intuition," says the biologist Stephan Harding, "but it is exactly those that give them ideas."

Too many numbers also drives out history – it gives us no sense of the different ways in which people measured in the past. It drives out creativity, locking away Keynes’ dark woolly monster of ideas. And it drives out morality too – leaving our poor beleaguered ethics committees desperately trying to measure themselves a coherent attitude to the frightening future of genetically-modified human beings, or whatever takes their place.  Where's your data, we will demand of them?  Where's your evidence?

And to get through the next few perilous decades, to look after each other, and solve the looming problems ahead, we're going to need all the judgement, intuition, history, creativity and morality we can possibly muster. So we have to make absolutely sure our tidal wave of data doesn't drive those things out.

My antidote to the tyranny of data is to ask the question the little boy asks in the Emperor's New Clothes.

Simple questions because they can devastate most political statistics. Yes, the carbon monoxide rate has reduced, but is the air cleaner? Yes, our local university professors have produced a record number of learned published papers, but is their teaching any good? Yes, the exam passes top the league tables, but what about the education? Are the children happy? Can they deal with life?

Data based on definitions is as vulnerable as the Emperor's New Clothes to the incisive, intuitive human question.  So is the utilitarian philosophy behind it.  Go ask...

Subscribe to this blog on email; send me a message with the words blog subscribe to dcboyle@gmail.com. 

When you want to stop, you can email me the word unsubscribe.

In praise of Michael Gove

I know it is de rigeur in some circles to curse the name of Gove. Two of the teachers at my children’s school danced a little jig together when they had heard he had been given the push in the reshuffle from the Department of Education. But I don’t share their delight.

This was, of course, Michael Gove’s problem. Not since John Patten’s tenure at the department has a Secretary of State for Education found himself quite so disliked by the profession – and I tell the strange story of Patten and the start of league tables in my book Broke.

I didn’t think that Gove got the emphasis right on everything, by any means, and it is bizarre the way that the paradoxes of education policy have been pushed to the same kind of extreme as the West Lothian Question.

Why are the only schools to have their curriculum prescribed the local authority ones? Especially if the main objection to local authority schools is their inflexibility. Why have local authorities been given so little leaway to find new school places, especially in areas where population is rising fastest?

But let’s leave all that on one side. Because Gove has been one of those rare things: a thinking, thoughtful, radical politician, and – as radicals ought to be – in constant conflict with the establishment. He needed to have cultivated more allies, in retrospect, but I still wish there were more like him.

The irony is that the elements of Goveism that most irritated the teachers – the emphasis on testing, the puritanism, the punitive approach to parents, the dour utilitarianism – may have had his approval, but were no more than extensions of the prevailing policy of his department for many governments past.

It was a dour, utilitarian place under Labour and it remained a dour, utilitarian place, with half the staff, under the coalition, but despite Gove. It was his misfortune that the department managed to shift the blame for this onto their Secretary of State.

But Gove was absolutely right on three issues, and these were not utilitarian at all.

He was right to emphasise the importance of diversity in education, right to champion the involvement of parents in starting schools and right to turbo-charge free schools. He was wrong to centralise their control and quite wrong to take them out of the local authority co-ordinating umbrella, but the basics of the free schools was overwhelmingly correct and overwhelmingly Liberal in the best sense – it can still be about diversity, self-help and non-conformity.

He was absolutely right in his emphasis on chronology in history teaching, and right to rescue history from the boulderised backwater where pupils have to study Hitler over and over again. You might choose different dates, but chronology is important to get across a sense of history and Gove was the scourge of the utilitarians here too.

He stood up against the alliance between the educational establishment and the online billionaires who believe that somehow you can have education without content. You don’t want the dull recitation of facts, but equally you can’t make education work without something to teach – relying on children to look things up on the iPads which the Pupil Premium has delivered them in such unnecessary numbers.

There are educationalists who think that education is just about process.  They are wrong, and without radicals and thinkers like Gove, the fear is we will also have government without content, the besetting sin of the British establishment in all ages. 

It wasn’t just that Gove was colourful and more interesting than the bland ranks of Conservatives (though he is). It was that he believed something and believed that, by sheer willpower, he could make it happen.

But there is a third reason too. He was rightly enraged by the failure of the secondary system a generation ago to lift children out of poverty.

Say what you like about league tables, and I do, they did reveal for the first time how schools were failing children. What the tables showed when they appeared for the first time in 1992 was that the national average of five passes at GCSE stood at only 38 per cent. Southwark Borough Council was bottom of the league, with 15 per cent (before the Lib Dems took over).

The most revealing comment of all at the time came from the head teacher of a school in Leeds where only two pupils had managed to scrape together five GCSEs: ‘We have a dreadful problem with truancy and discipline,” he said. “We have intrusions like motorbikes being ridden into school during the day while lessons are being taught.’

The very honesty seemed to demonstrate the scale of the problem, especially as he added that they were the best rugby league school on the country. So that’s alright then. Find out more about this in Broke.

I wouldn’t be surprised if pupils of that period didn’t bring a class action against their local education authorities. They certainly ought to.

Those days have gone in most parts of the country, though the league tables have become a kind of tyranny of their own. Even so, Gove has been fierce in his determination to drive standards for the poorest. That is why he’s such a loss to the Conservatives.

Subscribe to this blog on email; send me a message with the words blog subscribe to dcboyle@gmail.com. 

When you want to stop, you can email me the word unsubscribe.

The real scandal behind the sausage cartel

It is strange that Adam Smith is quoted so widely, not to say obsessively in the American economics departments, but his great warning about monopoly seems to have been forgotten:

"People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices."

This is, of course, a way of talking about the German sausage cartel, revealed today.  There was a discussion on the Today programme, full of the inevitable puns, which suggested that this was a blot on the reputation of German business (there have been cartels in other things recently).

I think rather the opposite.  The fact that these conspiracies against the public have been unmasked in Germany is a major positive.  If only we had competition authorities like that - but we don't: my only brush with the Competition Commission appeared to reveal an organisation staffed mainly by secondees from the European Commission, and with quite a different agenda.

Their interpretation of competition seemed to have little to do with breaking up cartels, and a great deal to do with building up European champions to take on the American semi-monopolies.

It was this confused body which conducted the recent flawed reviews of the UK groceries market.

I am coming to the conclusion that the blindness of the UK establishment about monopolies has a great deal to do with the failure of the Liberal Party to play a greater role in the second half of the twentieth century - and, when they did come to play a role, they seem to have forgotten their original economics.

Competition, competition, competition, plus diversity.  How could we forget?

So here's my take on the sausages.  It is bizarre, isn't it, that companies are not allowed to collude with each other about prices - yet they seem to be allowed to buy their competitors almost at will.

The UK's abject competition authorities allowed Waterstones's to buy Ottakars, almost their last high street competitor.  Fair enough, maybe that was the only way of preventing collapse - though I doubt it.  But then to let Amazon buy their only UK online competitor in the books market, the Book Depository.

So we are subject, over and over again, by conspiracies against the public by the back door.  And all because Liberals and liberals forgot the central economic lesson of Liberalism.

This is how Adam Smith continued the paragraph:

"It is impossible indeed to prevent such meetings, by any law which either could be executed, or would be consistent with liberty or justice. But though the law cannot hinder people of the same trade from sometimes assembling together, it ought to do nothing to facilitate such assemblies; much less to render them necessary."

Quite.

Time to double London's congestion charge

The American mapmaker Rand McNally launched their groundbreaking Places Rated Almanac in 1981 and it changed the whole argument about cities.

It coincided with research that showed the most attractive cities for investment were not, actually, the places with no regulations and tax breaks.  They were the places where the CEOs of companies wanted to live - and they wanted clean air, green space, theatres, good schools for their children and so on.

It marked the beginning of the Civics movement in the USA and it rescued cities like Pittsburgh, where - in the 1970s - the streetlights had to be kept on in daytime because of the polluted air.

The bottom line was this.  Cities which allowed themselves to get polluted because it was somehow a side-effect of thrusting economic success were completely deluded.  To succeed in the long-term, cities had to be good places to live.

So I find it pretty extraordinary that, three decades or more later, the neanderthals appear still to be in charge of London.

I'm not sure I believe the hype that the London air is the most polluted in the world - it would take a lot to beat some of the Far Eastern cities - but it is still pretty bad.

Especially in the last few weeks, when all my friends appear to have earned themselves sore throats and chest complaints for working in London.  As I wander around the centre of town, I am staggered that we have allowed almost permanent traffic jams to take up residence at key roads and junctions.

Why have London's bosses failed to learn the lessons?  It is partly a deeper malaise: the Department of Transport has shifted pro-rail, but they have failed to learn a more important lesson - that the key to cutting traffic is paradoxically to reduce, not increase, the space for cars.

Unlike many cities in the UK, London's transport users do have choices - most of them - and they will use them if that traffic becomes intolerable for them.  But that is a decision about time, not a decision about pollution.

But those choices mean that, unusually, there is a policy option that would help solve the problem and raise considerable sums of money.  It is time to double the Congestion Charge.

What are the arguments against?  That poorer people will be unable to drive in London.  They already can't - have you seen Westminster's parking charges?

That some people really need to drive across London.  Definitely, but give them discounts - and give them the road space to let them do it.

The future of London depends on it being green, clean, pleasant place to breathe.  Nobody will bring their children to live in a polluted hothouse.  Nor will the city thrive if its leaders suffer from British-Establishment-Disease (definition: the inability to make a difference; the habit of making occasional empty gestures towards problems).

But personally, I'm not waiting for a solution any longer.  I'm off.

The World Cup and the Curse of Oil

I share a house with two small boys so I have been watching and listening to more football these last few weeks than I have done since I was 12, and Chelsea was playing and replaying Leeds or the FA Cup (1970, if you must know).

It has made me wonder a little why the most lucrative, expensive and money-laden nation on earth - when it comes to football - should produce such a less than successful team.

There has been rather too much involvement by very old-fashioned economists in the football debate.  Goldman Sachs calculated that Brazil would win, so - for that reason at least - I enjoyed their 7-1 defeat, as a blow against the idea that everything can be reliably measured.

So why are England not better?  I think it has something to do with the phenomenon known as the Curse of Oil.

The Curse of Oil is the peculiar paradox which means that the nations rich in raw materials, and oil in particular, find that it does them no good. “What makes her poor is her wealth,” said a 16th-century Spanish economist about Spain, awash with gold and struggling with the effects of crippling inflation as a result. A similar disaster overtook Peru during the guano boom (to 1870) and Brazil during the rubber boom (to 1920).

Nothing excites governments more than suddenly discovering, as the British did in the 1960s, that they have oil on their back doorsteps. The rush of black gold from the seabed or deep in the earth is enough to make the owners of the rights believe that their troubles are over once and for all. Actually the opposite is true, which is why Juan Pablo Perez Alfonzo, the former Venezuelan oil minister and OPEC co-founder called oil ‘the devil’s excrement’.

There are exceptions to the rule, like Norway and Malaysia which have used their oil revenues to diversify their economies, but generally speaking the countries that are most dependent on oil wealth do worse over time, and the countries least dependent on it do best. Strange isn’t it.

Oil lulls nations into a false sense of security. They believe the wealth will cushion them against the need for tough decisions and investment. They believe energy will always be cheap, so never have to innovate to save energy. Meanwhile the inventions and efficiency that result from a shortage of energy go to their oil-poor competitors.

The UK’s own version of the Curse of Oil was a ruinously high pound. It means the world can no longer afford your products, and your factories start closing one by one, as they did in the UK during the peak oil production periods in the 1980s.

The Curse of Oil is the proof that you can't just measure the money and assume that is a measure of quality, success or well-being.  Quite the reverse: too much money can be disastrous, as it was for Spain in its Golden Age.

And look what it does in football.  Because the money sloshes around English football, they buy the best players in the world and forget to develop their own talent.  Soon they forget they have talent at all.

Result: we don't make it out of the groups in the World Cup.

The great seed-sharing tug-of-war

I have to confess, I find the new campaign websites pretty im-possible.  I was involved in the early discussions to launch 38 Degrees, but I've ending up chewing the carpet in frustration about their infuriating naivety and ignorance about what is actually happening at Westminster, which wastes huge amounts of energy, mainly theirs and that of their supporters.

Avaaz is, in some ways, even more irritating.  They pick the right causes, but they don't seem to have thought through the basics of transparency.

Take their latest campaign to crowdfund a global non-GM seed exchange.  Basic politeness suggests that it might make sense to say how much money is needed in total, and how much has been raised.  Nothing - and no replies to my requests for an answer.

There may be a good reason for the silence, in which case it might make sense to say so.  Otherwise it is arrogant and unimaginative and probably counter-productive.

So why did I carry on regardless and donate?

Because the issue of who owns the world's seeds, and the genetic heritage that lies behind it, is so important.  And the idea of a seed bank which allows small farmers all over the world to exchange seeds, including the whole diversity of the planet's genetic heritage - and without having to shell out a cut to the monopolists - is potentially a game-changer.  If the monopolists allow it to happen (they don't like competition).

In fact, here is an opportunity for the two models of the free market to go head to head.

Model 1 (actually Model 1B, but let's leave that aside) suggests that food production will be supported most effectively by large companies which have efficiencies of scale and which can develop and own new varieties of seeds which they can control.

Model 2 suggests that food production will be supported most effectively by supporting the small farmers who do it most efficiently, and by a system of sharing that allows them to keep more of their own profits.

Model 1 means concentration; Model 2 involves sharing and diversity.  Which works best?

In 1845, there was a famous tug of war between the screw-propelled HMS Rattler and the paddle-powered HMS Alecto.  The Rattler won convincingly, towing its opponent backwards at a rate of two knots and the paddle went out of fashion.

If we can get this right, then we will have Monsanto versus Seed Bank Sharing tug-of-war, and we will now one way or another, which is the most effective.  When the Seed Bank tows Monsanto backwards at two knots, we will know something important has happened.

But Avaaz has to come clean about some of the details - how much money is needed, who will be responsible, who will own it, when will it begin?  They just need to take their supporters seriously.  Because if they can't take the punters seriously, how are they going to take the formidable and well-resourced opposition (Model 1) seriously?

Why don't we ever commission entrepreneurs?

It has been a brilliant summer so far, hasn’t it? It reminds me of the 1990s, when I remember – day after day – getting up, putting on my shorts and sandals and heading off to Cowley Street to edit Liberal Democrat News (no, sandals weren’t compulsory).

But I do have one local difficulty. The lane at the end of my road, which gives me access to my local area and my childrens’ school, has become flooded and boggy.

In the rain, the water pours down and into the street. Now, in the summer, it is at least passable, but I can still see the trickle washing through into the bog.

The water is still running despite weeks of sunny days. A development next door has disturbed one of the many springs in the area, and it doesn’t stop (these are, after all, the hills that gave us the Beulah Spa in the 1840s).

I have seen water board investigators there. I’ve seen local authority investigators there. I’ve had the same conversations with them many times. I was finally assured in early December by Croydon Council that I would get an answer within three weeks. I’m still waiting.

But here’s the point. I'm told the reason why I’m still waiting is that the company the council wants to tackle the problem has been asked to tender for the work, and is dragging their feet. Still no tender. And so we wait.

Now this is very peculiar, and yet somehow familiar. The whole justification for giving up on direct labour organisations and commissioning private companies to do the work instead is that it was supposed to be more flexible.

If one company couldn’t or wouldn’t do the work, and at a reasonable pace, quality and price, there would be others that could be employed instead.

That's the theory. The practice is quite different. We wait, six months now, for a company that clearly doesn't want the work to tender for it.

For me, it is yet another example, not of the perils of private contractors (that's another question) - but of the failure of government at every level to reap the benefits they should from the flexibility of an open market.

Instead of privatisation, what we actually have is a muddle of interlocking and contradictory regulations – from preferred supplier lists to minimum size limits – which actually get it her way of open markets, and do so in the name of open markets.

I have written before about how open market regulations at the European level get in the way of so much activity happening at a local level. The same thing happens at local level too.

This is bizarre, and is so far from what passes for a debate between state and private – which takes place on a different plane, about a different planet. We consequently get the worst of both worlds – no security, little quality control, ridiculous delays and extra costs, and all in the name of enterprise and open markets. And no competition.

At a European level, we know that at least some of the regulation around the single market is there thanks to lobbyists from the biggest companies, and is designed partly to squeeze out competition from the smallest.

We are also assured that the new EU/US trade negotiations (TTIP) will result in growth of 0.5 per cent in each EU country, with 545 euros per household pouring through the economy.

Is there really any evidence for this? Is it really going to do more than shift existing spending into the pockets of the biggest and least effective companies and do more to crowd out the genuine entrepreneurs?

It is true that the whole purpose of TTIP is to extend the ability to export services to smaller companies. But there is the problem. At local and international level, open markets appears to mean more definitions, processes, limits and restrictions. It is the precise opposite of what it is supposed to be - in fact single market regulations, inspired by the UK, are the source of the most spectacular EU mythological regulation stories.

What can be done about it? I don't know. But I'd start with a bonfire of regulations and procurement processes at local level.

Heavens, you might say! How are small companies and social enterprises ever going to get commissioning contracts? But they are not getting contracts under the current system.

Something must change - but for that to happen, we need a dose of ideology that can see open markets clearly. That means rediscovering the idea of free trade from a Liberal perspective.