Decoding Official Secrecy in the Age of Data-Mining

Professor Matthew Connelly, Department of History, Columbia University

Co-presented with the Laureate Research Program in International History, the University of Sydney

13 May, 2014

The scope of official secrecy in the US is growing exponentially. The sheer scale of the national security state, the growth of electronic media, and the power that still comes from compartmentalizing information means that Washington is only releasing a tiny fraction of the classified information it produces annually. Millions of secret documents are piling up, and millions more are being destroyed or deleted, raising doubts about how we will ever be able to reconstruct the past and ensure government accountability.

But historians are now teaming up with data scientists to analyse the documents that are being released, as well as the metadata for still-classified records. Data-mining, in other words, is a tool that citizens can use to track government activity, and not just a tool for government to surveil citizens. It may soon be possible to make out the broad patterns of official secrecy, attribute authorship to anonymous documents, and perhaps even predict the content of redacted text. But the political and ethical questions remain: what does the public need to know, and when do they need to know it?

Professor Matthew Connelly

Professor Matthew Connelly works in international and global history at Columbia University. He received his B.A. from Columbia (1990) and his Ph.D. from Yale ( 1997). His publications include A Diplomatic Revolution: Algeria's Fight for Independence and the Origins of the Post-Cold War Era (2002), and Fatal Misconception: The Struggle to Control World Population (2008). He has written research articles in Comparative Studies in Society and History, The International Journal of Middle East Studies, The American Historical Review, The Review francaise d'histoire d'Outre-mer, The Review Francaise d'histoire d'Outre-mer, and Past & Present. He has also published commentary on international affairs in The Atlantic Monthly and The National Interest.