UK Intelligence Needle Lost in Data Haystack

Douglas Crawford

Douglas Crawford

June 16, 2016

Newly released documents obtained via Edward Snowden definitively prove what common sense has told us all along. Hoovering up vastly more data than can be meaningfully analyzed actively hinders the ability to find criminal and terrorists. As far back as 2010, UK security agencies were suffering from a data haystack that the draft of a top-secret report admits could put lives in danger.

The Security Service… can currently collect (whether itself or through partners such as NTAC) significantly more than it is able to exploit fully. This creates a real risk of ‘intelligence failure’ i.e. from the Service being unable to access potentially life-saving intelligence from data that it has already collected.

Just last week the UK’s House of Commons passed the Investigatory Powers Bill, so the publication of this report could hardly be timelier. The so-called “Snoopers Charter” grants the UK government unprecedented powers to spy on the personal web browsing history of every British citizen.

It also grants the government legal power to spy on all international data that passes through the fiber-optic cables that terminate in the UK. This constitutes a very high percentage of the world’s entire internet data.

It stands to simple reason that the bigger the haystack, the harder it is to find a needle. By collecting ever larger troves of indiscriminate data via mass surveillance operations on the general public, security services are actively hindering their ability to detect criminals and terrorists.

As The Intercept observes, three years after this report was authored two domestic terrorists brutally murdered off-duty British soldier Lee Rigby. They did this despite being well-known (.pdf) to MI5. The indigence agency had obtained records of a phone conversation one of the attackers had made to a known al-Qaeda member in Yemen. It also knew of a social media post which described the pair’s intentions in detail!

A follow-up secret report was published in March 2010. This explicitly spelled out the disparity that existed between the vast amount of raw information available to MI5, and its ability to usefully process it,

There is an imbalance between collection and exploitation capabilities, resulting in a failure to make effective use of some of the intelligence collected today. With the exception of the highest priority investigations, a lack of staff and tools means that investigators are presented with raw and unfiltered DIGINT data. Frequently, this material is not fully assessed because of the significant time required to review it.

Silkie Carlo, a spokesperson for civil liberties group Liberty, told The Intercept that,

Intelligence whistleblowers have warned that the agencies are drowning in data — and now we have it confirmed from the heart of the U.K. government. If our agencies have risked missing ‘life-saving intelligence’ by collecting ‘significantly’ more data than they can analyze, how can they justify casting the net yet wider in the toxic Investigatory Powers Bill?”

Despite this, the government is pushing forward with plans to collect ever more data.

A wider “haystack problem”

Although both reports concentrate on MI5’s intelligence gathering capabilities, the problem affects all UK security services. This is because all UK spy and law enforcement agencies’ relationship with British telecoms companies are mediated by GCHQ, which covertly intercepts phone calls, text messages, and internet data sent using a program code-named PRESTON.

A 2009 study found that PRESTON hoovered up more than 5 million communications over a 6 month period. Yet despite these intercepts being specifically targeted (i.e. not just blanket surveillance), only 3 percent of them had been viewed!

Preston Study shows SIGINT needle lost in data haystack

A very small proportion of the Preston Traffic is viewed. This is of concern as the collection is all warranted.

So why make the haystack bigger?

The more data security services collect, the harder it is to find the needle. Sounds obvious, doesn’t it? And these newly leaked documents confirm the point. So why is the UK government (which is hardly alone on this issue) so determined to collect even more data, and therefore make the haystack even bigger?

Sometime in the late 1780s, renowned English philosopher and social theorist Jeremy Bentham came up with the idea of the Panopticon (literally “all observed”). It is a prison building designed to allow a single watchman to observe all the inmates.

Of course, such a watchman cannot keep an eye on all the inmates all the time, but the fiendish genius of this design is that the inmates know they can be watched at any time. Knowing this, the inmates would be forced to behave at all times as if they were being watched if they wished to avoid disciplinary measures. Bentham hailed the Panopticon as,

A new mode of obtaining power of mind over mind, in a quantity hitherto without example.”

The Panopticon has since become a powerful metaphor for the “chilling effect” that surveillance has on free speech.  As George Orwell well understood, when people fear that anything they do might be watched at any time, they will behave accordingly,

“You have to live – did live, from habit that became instinct – in the assumption that every sound you made was overheard, and, except in darkness, every movement scrutinized.”


GCHQ has been mass spying on our communications since at least 2001. With its current high-profile push to “formalize” what has hitherto been going on in secret, however, the UK the government is sending a very clear message to the public. You are being watched.

There exists plenty of evidence to show that when people believe they are subject to mass surveillance they behave accordingly. They will self-censor what they say to others, and what they look at on the internet.

If the government’s real aim is to catch terrorists and suchlike, then it would be spending resources on finding ways to narrow down its search in order to sift more effectively through relevant data. Instead, it wants access to ever more troves data that it cannot meaningfully use or analyze. As we have already noted, the bigger the haystack, the harder it is to find the needle.

Given this basic fact, it becomes obvious that the government’s desire to spy on everyone and everything has nothing to do with finding terrorists. It does, however, have everything to do with building an Orwellian state aimed at exerting power over an intimidated population…

Douglas Crawford

I am a freelance writer, technology enthusiast, and lover of life who enjoys spinning words and sharing knowledge for a living. You can now follow me on Twitter - @douglasjcrawf.

4 responses to “UK Intelligence Needle Lost in Data Haystack

  1. Excellent post and very true.

    The panopticon effect will become even more pronounced as artificial intelligence develops and enables the grunt work of shifting through the haystack of data to be done by machines. People will really change their behavior when they start getting individual attention for their online actions like Winston Smith got when wasn’t performing the daily physical jerks with appropriate gusto.

    I mean, if terrorists are still posting their intentions on the clear net without a thought, it seems unlikely that regular folks are really minding what they do online. AI will change that.

    At the same time as making citizens fear the panopticon they will want to keep people from moving all their communications to the deep web or i2p or wherever. Better to slowly make their presence felt with more innocuous slaps on the wrists to individuals than to make a big show and dance of their surveillance early and possibly spark a popular movement that rejects using networks that can be monitored.

    But yes, obviously anyone who is up to no good is going to work around this panopticon and the only people inside will be the proles that never bother to open their eyes. It won’t take long for the dumb terrorists to wise up and then the only use for this surveillance will be to make an Orwellian state. That and creating a treasure trove for hackers I guess.

  2. i could easily filter out the data hastack. filtering is cakewalk, besides the fact that means i could get the fbi to ‘ease-off’ the backbone of the internet, therefore unthrottleing all the poor ‘tier 16’ internet users on branch 3.

    i can tell u specifically how to filter for actually relevant data. and where the true cyber criminals lurk. i can assure you they have no reason to use a vpn, much less an ip address.

    1. Hi gunmaker,

      It is true that “true cyber criminals” use the dark web (e.g. Tor hidden services) instead of trying to hide themselves on the regular internet.

Leave a Reply

Your email address will not be published. Required fields are marked *

Exclusive Offer
Get NordVPN for only