Shit Happens and Big Data
When they found the body in the peat, one contingent went off to find Gil Cunningham to handle matters while the other went off to grab the witch they knew was responsible for the death. But Gil Cunningham, hero of a series of novels by Pat McIntosh, is a bit more modern than the 15th century in which the tales are set. He makes an examination, releases the witch, takes the body back to the manor for a closer inspection, and undertakes a larger investigation.
Unlike in Gil Cunningham’s sphere, in the 15th and 16th centuries, inspired by an expose of witches called Malleus Maleficarum written by two monks, French and German witch hunters killed between 60,000 and 100,000 accused witches, according to Steven Pinker’s The Better Angels of Our Nature: Why Violence Has Declined. Pinker cites two pressures which contributed to the waning of superstitious killing, one “an increased valuation of human life and happiness”; the other a recognition that some things happen due to “impersonal physical forces and raw chance,” or as Pinker more succinctly puts, a recognition that “shit happens.”
In Gil Cunningham’s 15th century Scotland, someone needed to be found guilty of the man-in-the-peat’s death, and found soon regardless of true guilt, in order to appease the forces that be and restore order. These views are not far distant from a past where human sacrifice was formally engaged in, in order to appease the gods and maintain the order of the universe. It was also convenient that the “guilty witch” in the story was part of a different community, a different tribe if you will, so that the guilt and punishment could be made to fall on others. Since those days, there has been a dramatic decrease in violence, as Pinker details, consonant with changing views and social structures. As society has moved forward, extended commerce has made partners out of potential enemies, and literacy and the application of reason have extended our understanding and allowed us better to take the perspectives of people unlike ourselves.
Still, it is often hard to break free from the notion that there is a cause for everything. Nicholas Wade, in his book A Troublesome Inheritance: Genes, Race and Human History argues that Europeans developed more open nation-states starting around A.D. 1000, that this caused selection for genes in the population which promoted more trust and social openness, which reinforced the ability to build more trusting and hard-working cultures and nations. This genetic-social development did not take place in Asia and Africa, leaving Europeans with genes better suited to a successful modern society. In his review, David Dobbs takes issue. The sentence which most caught my attention was, “While warning us to avoid filtering science through politics, he draws heavily from conservative historians who minimize the roles played by political power, geographic advantage, momentum, disease and dumb luck.” “Dumb luck.” “The impersonal forces of nature.” Not everything that happens is pre-ordained. Wade, the reviewer might say, is stuck on causation. “The result,” the reviewer asserts, “is a deeply flawed, deceptive, and dangerous book.”
So how does big data fit into this picture? Viktor Mayer-Schonberger and Kenneth Cukier repeatedly make the point in Big Data: A Revolution That Will Transform How We Live, Work, and Think, that applying big data to problems allows us to come up with meaningful and actionable correlations independent of any concern for causation. This takes us a further step away from the idea that there is some supernatural reason for everything. Not only are there “impersonal physical forces” responsible for some of the things which happen in life, but even when there is a cause, it’s often not worth our while to seek it, since we can accomplish so much merely by applying data and discovering correlations.
Pinker often picks up on the theme, in one case when he investigates and finds that wars break out randomly and follow a power-law distribution. Power law distributions are interesting because they look the same over a vast range of values rather than heaping around a particular set of values as with a bell curve. Examining war in such a statistical fashion could allow us to discover things we might not have otherwise if we merely examined them in a historical/political context. Some of the items that Pinker notes are that the same dynamics which apply to smaller engagements, threats, backing down, bluffing, engaging, escalating, fighting on, or surrendering, apply to larger engagements. Also, since power law distributions have long tails, meaning in this case that larger wars are less frequent but not astronomically unlikely, and that a conflagration much larger than two world wars we have experienced is not out of the question.
Unlike the examples in Big Data, where the goal is merely to find correlations that are actionable, Pinker does wish to understand the dynamics, from psychology, politics, and technology, which would cause wars to break out on the power law distribution, notes work in other fields on similar investigations, but admits that there is not enough data on wars to come to an answer.
The Big Data authors are not unaware of the dangers of big data. If we can predict, we can take measures to minimize or maximize the outcomes we are predicting. Much as in Minority Report, we could predict who will commit crimes in advance and incarcerate them preventatively. That this counters the notion of free will is certainly a concern. It also feels spookily like knowing who’s a witch and responsible for the dead body in the peat. Acting on our predictions is much like assuming causation and returning us to a situation where shit doesn’t happen, but all is determinable.
The world has gotten less violent, in part because, recognizing that the impersonal forces of the universe are responsible for much of what happens, we don’t feel as much need to seek retribution when bad things happen. This signals an increase in the level of abstraction through which we view the world, and signals a turn away from emphasizing causation, and toward one on correlation. This is the foundation for the big data revolution which Mayer-Schonberger and Cukier describe, and is part of the foundation of Pinker’s approach to analyzing war and peace.