The imperative of openness for data society

Aficionados of Isaac Asimov’s Foundation series will like this article from MIT Technology Review. In the article we can read bout how physicists have developed methods that allow forecasting, much like weather forecasting, of purchasing decisions. And the way they do this should sound familiar to Asimov fans:

Here’s how they do it. These guys think of humans as if they were atoms interacting with each other via three different forces. The first is advertising, which they think of as a general external force, like a magnetic field. The second is a word-of -mouth effect, which they model as a two-body interaction. Finally, they think of rumours as an interaction between three bodies.

The possibility of predicting the systems as the number of forces working on the atoms increase is of course the really tricky question. Asimov also used the analogy of a gas and establishes a number of important theorems that form the basis of psychohistory. One of the most important ones, but one that I believe is often excluded from enthusiastic discussions of how we could use data sets to predict different things, is the second (or third, accounts vary) theorem of psychohistory. It says:

that the population should remain in ignorance of the results of the application of psychohistorical analyses

I have seen people react differently to this idea, that the usefulness of a predictive system declines with how well the predictions are known, but most reactions seem to want to trivialize the problem. Yes, people will say, but let’s try anyway. I think that is a good attitude, but it is theoretically interesting to examine what it would mean if we move down the road that research like the one referenced in the MIT Tech Review opens up.

In one sense psychohistory is a manipulative tool. A society predicted loses some, perhaps all, of its democratic nature, and we move from a free market to a more planned economy. Hayek famously argued that the knowledge coordination problem is not only best solved, but only solvable, through markets, but there are those who believe that modern technology proves him wrong — that we could predict and plan society much better now with the large data sets that are accumulating everywhere in our society. They are not necessarily wrong, but their conclusion, per the second theorem, depends on those data sets and predictions being kept secret from others.

But, they argue, that is fine. Because the value of living in a carefully predicted and planned society is larger than the value of living in a society where everyone has access to the data used for the predictions and where these predictions are thus rendered useless. Their arguments will range from national security, health, economics to social equality.

Let’s assume for a moment that the complexity facing any attempt at psycho-history is not such that it renders all attempts futile. That we could in fact develop social predictions with high accuracy through the use of large data sets, and that we could act on them and plan our societies accordingly. The question we then have to ask ourselves is this:

Does the value of predictability trump the value of openness?

There is an assumption here that is worth highlighting. And that is that for a democracy to remain open it can not be predictable by only a few. That is a complex and perhaps provocative assumption that I think we should examine. I believe this to be true, but others will say that our democracy already is predictable, in some sections and instances, only to a few and that they build their power base on that information asymmetry, but that it is reasonably open still. Maybe. But I think that those asymmetries are not systematic to our democracy, but confined to those phenomena, like stock markets, where they are certain to be important, but where they also do not threat the nature of democracy as such.

In summary, if we share the data and allow everyone to use it, then predictability goes down. And I think it decreases fairly fast. So in a simplistic graph:

This leads to the guess that in the information society, open access to data is not only a great way to encourage new entrepreneurship, it is actually also a safe-guard for an open democracy. Governments are still, by far, the largest data holders, I think, and the data they collect is very suitable for social predictions. (And yes, companies should do this too. We do, and others do too, through tools like trends and insights, and we believe that the world is better for it, with lots of research flowing from that shared data).

I remember reading Asimov, and re-reading him, with awe. I loved the idea of psycho-history, and though that if only the math-geeks were in control then things would turn out just fine. I thought Asimov himself was a prophet. Turns out that he was quite sceptical, and often talked about both the advantages and dis-advantages of being able to predict societies and plan them.

In fact, in Asimov’s later writings it turns out to be a robot that encourages Seldon to develop his science into a social means of control. As we all know, the robots in Asimov operate under the three laws, and they want to reduce harm to humans. A predictable society would allow for that to be done as carefully as possible, but it would also curtail some of the human spirit, creativity and holy insanity that make us human.

If there is a conclusion here it seems to be to explore the amazing value of data under the imperative of openness to the the full extent possible to ensure that our societies gain from this new, fantastic age of data innovation, discovery and exploration that we are entering into, but never compromise on that openness in the pursuit of macro-social predictability.

Dela gärna


  1. said:

    Thank you for emphasizing the ethical point of view on data gathering and societal prediction – it´s easy to miss out when excited about the new possibilities!

    Also thank you for the the links to the very interesting research. I notice that they all seems to be based on search queries as indata – you don´t happen to know if there is any similar research out there based on data collected by Google, but based on psychological text analysis, such as this small experiment over at Facebook?

    December 7, 2011
  2. nicklas said:

    @mattias – not that I know of, but I will keep my eyes open!

    December 8, 2011
  3. Pär Lannerö said:

    “we could predict and plan society much better now with the large data sets that are accumulating”

    In other words, the data collected by, for example, Google could be used as an excuse to give communism and 5-year plans a new chance? :)

    December 8, 2011
  4. nicklas said:

    @par I am certain some people would, but it requires the data sets available in the public sector. That is why I stress that we and other companies are open with the data sets (see the research links). It sounds weird, I know, but the argument for planning is very tempting to some (especially those who think that they understand things much better than the “average” person)

    December 8, 2011

Kommentera gärna!