The Problem With Papers
Colin Frayn, 23-24 October 2006
A few months ago I found myself sitting in a conference, confused by the level of needless mathematics that seemed to be creeping into every talk I attended; an addition that was not only unnecessary, but which actually confused the valid points that the presenters were trying to make. I penned the following paragraph in the margin of my lecture notes:
" It seems to me that an extraordinary amount of maths, complex vocabulary and acronyms are required to encode [with no extra information content] basic common sense, or to compensate for a lack of it. If you have intuitive understanding, vision and imagination, then you can achieve world class results in virtually any area of computation with no mathematics whatsoever. And what's more, you can explain your work in language fully accessible to the general public. "
This short essay is about a subject that is very close to my heart as a scientist, but which seems to be very greatly neglected within Science. That subject is 'teaching'. In this short discussion, as with all my writing, I will distinguish between 'science', with a lower-case 's', meaning 'the concept of obtaining knowledge through scientific method', and 'Science' with an upper-case 'S', meaning the set of institutions and the researchers within those institutions which make up the majority of scientific research worldwide. I think this is a necessary separation, as will become clear later.
The word 'science', like so much of the English language, comes from Latin roots. In this case, originally from the word 'scientia', meaning 'knowledge'. But the etymology goes back way further than that, to the original stem 'scindere', meaning 'to cut or divide'. Knowledge allowed people to make divisions or distinctions between classes of things. Science, then, was (and is) the acquisition of knowledge with an overall purpose, and using a well-defined method. Science is all about increasing knowledge in a way that can prescribe some degree of confidence to that knowledge, and which allows for a rigorous process of criticism and analysis.
Scientific method evolved from the earliest logical thought as a process by which scientists could investigate phenomena in the world around them. The scientific establishment arose as a manifestation of that process in people, bricks and mortar. Universities and private research laboratories began to spring up with the Renaissance, and their growth has continued unabated ever since. Part of that same scientific community is the mechanism for the necessary exchange of ideas and information between individual scientists, which permits the spread of knowledge as rapidly as possible for the benefit of the human race.
Such a communication system can also double as a repository of all scientific knowledge, as it gives a history of all scientific thought, experimentation and argument taking place between knowledgeable individuals. That is, professional scientists. As such, and viewing its importance in the whole process of scientific discovery, it is therefore necessary that the communication process subject itself to a rational system of control. That system should be able to ensure that information entering into our species' repository of knowledge is in accordance with the massive volume of knowledge going before or, in the case of a new and potentially contradictory discovery, that it submits itself to the burden of a necessary and sufficient degree of proof.
So, scientists aiming to transfer their discoveries into this repository need first to present their work for review in order to ensure that it meets the necessary criteria. Who better to evaluate the validity of a work, and indeed the contribution it makes within the general body of scientific knowledge, than other scientists working in the same area of research? Hence the system of peer review was initiated, which forms the foundation for the incremental expansion of human knowledge.
The role of an academic is twofold: to develop knowledge and to distribute it. Either alone is useless. Scientific discoveries must therefore be spread, and this usually involves the publication of scientific papers, usually in journal papers, or orally at a conference or seminar. The speed of scientific progress depends not only on the work of the individual researchers, but also on the efficiency with which they are able to upload their new distinctions into the consciousness of all other relevant researchers. And the degree to which science serves humanity can be measured by the volume of these distinctions which make it past the walls of academia and into the public sphere. More importantly, into the lives of those who could most benefit from the advances that this new learning provides.
So, given the importance of communication within Science, one might envision a system in which information is disseminated in the most efficient and clear manner possible, so as to accelerate this learning process to its maximum speed. But sadly, as you may have guessed by the title of this essay, the reverse is true. Academic communication is fraught with incentives which work against the efficient transfer of information, and against the development of high-quality research.
But it's not just lesser names in Science who are guilty of this problem. Claude Shannon, one of the founders of information theory, wrote in a letter to a friend of his that he should use the term 'cybernetics' for the discussion of a new field of research, "because nobody knows what it means. This will put you at an advantage in arguments." In a world where perceived intelligence is the key to a successful and glittering career, the incentive is to magnify one's perceived intelligence, often at the expense of clarity and progress. This is the first misaligned incentive of Science as it stands today.
I spent most of the last three years studying the field of evolutionary computation, where solutions to a problem are evolved in response to a set of global incentives, always pushing them in one desired direction. However, one of the most important lessons that anyone can learn from such a system is that life will always find a way to exploit any situation unless the utility function is precisely aligned with the objectives that one wishes to achieve. In the real world, the only objective is to remain alive and become a successful ancestor, so life finds all manner of ways of achieving this from human beings, gorillas and dolphins right down to seaweed, mushrooms, bacteria and viruses. In a computer we often arrive at a similar scenario.
Misaligned incentives will push people away from the direction in which they are required to develop. The communist system failed because it removed all positive incentives, and only achieved any degree of order whatsoever by the use of brutal force. Governments tasked with increasing the numbers of young adults attending University, will do so not by increasing the standard and quantity of high school teaching, but rather by relaxing the standards, and redefining 'University' to include a number of institutions previously considered otherwise. Governments are incentivised to remain in power, and will therefore do whatever they can to fulfil this goal first and foremost, even if that means neglecting longer-term policies which could pay off in the future, but which will make re-election less likely for the short-term. Misaligned or ineffective incentives produce systems which do not fulfil the purpose with which they were founded.
The second misaligned incentive of academia is similar, namely that an academic is strongly judged on the quantity of papers he or she produces. As it is almost impossible to judge the quality of such papers in a fair and reliable way, the only reasonable criterion that one can apply is simply the quality of the journals in which they are published. A very senior academic once mentioned to me the phenomenon of 'salami slicing', where otherwise reputable Scientists who have produced a good piece of work, try to subdivide it into as many smaller chunks as possible in order to increase the size of their publication record. It is a very common trick. Friends add each other's name to the author list in papers merely in recognition for a few useful discussions, with the aim of expanding their publication record yet further.
So we have these two incentives, introduced with good reason in order to verify the contribution that academics are producing to their subject. In this age of accountability, we are being forced to justify our existence in an ever increasing number of ways. I can't immediately think of any one system for doing so which is not open to abuse, and which does not produce an outcome at least partially misaligned with the desired overall goal.
So what is the system produced by this over-reliance on peer-review? Well, it is a system that encourages academics to produce quantity rather than quality; to spread a meagre amount of work over as large a number of publications as possible, thus seriously diluting the information content of the global repository; to obfuscate their publications with mathematics and complex phrasing simply to give the impression that the work presented is more substantial, and hence reducing the efficiency with which such information is conveyed; to build up an ivory tower of Science which is seen as inpenetrable to the outside world, and which necessitates specific government efforts to produce a largely apathetic public interaction. Why should this be, when the products of modern science are at the very core of our everyday lives? Why is it that the public views Science as such a distant, myopic and dusty subject? Why is it that public understanding of science is so poor, in an age of global information transparency through the Internet? I suggest that the above would go a long way towards answering these very questions.
Science is not hard. Einstein explained relativity in a way that even a primary school child can understand. He did this using imagery that people intuitively recognised, in order to make use of the brain in ways for which it evolved. This is how he discovered the theory in the first place - by 'thought experiments', using analogy and visual metaphor to investigate complex processes. Einstein only published around fifty papers in his lifetime - a figure that would probably get him kicked out of most top Universities nowadays - but he certainly made them count. This one man, in just a handful of works produced during his early life, revolutionised all of modern physics and laid the foundation for innumerable twenty-first century technologies.
When I teach natural Computation, I do so without any maths whatsoever. If I try to explain this teaching style to experienced lecturers, as I have on occasion, I am met with incomprehension and dismay. Yet I have received enormously positive feedback from the students themselves. Some say that they had sat through terms of lectures on a subject and followed very little, whereas within a few short hours of being taught in a way that their minds had evolved to understand, they developed a far better grasp of the subject than ever before. I say this not to boast, but only to demonstrate that it is possible, and I would argue vital, to teach subjects properly. And the only reason, as far as I can see, why this is not done in general, is simply because of the reaction that I so often get after a lecture: "Is that all there is? But it's so simple!"
Yes, it really is simple. But it takes a true understanding in order to show someone just how simple it all is. And it takes the courage to accept that people may no longer view you as the omniscient genius that perhaps you would have liked, but instead they appreciate you more because you have given them something far more valuable than awe and admiration: you have given them knowledge.
And after all, that is what science is all about.