This report is also available in French, German and Arabic.
Please read on for the Introduction.
This paper uses automated text analysis – the process by which unstructured text is extracted, organised and processed into a meaningful format – to develop tools capable of analysing Islamic State (IS) propaganda at scale. Although we have used a static archive of IS material, the underlying principle is that these techniques can be deployed against content produced by any number of violent extremist movements in real‑time. This study therefore aims to complement work that looks at technology‑driven strategies employed by social media, video‑hosting and file‑sharing platforms to tackle violent extremist content disseminators. In general, these platforms aim to remove material produced by terrorist and hate organisations unless such material is disseminated in very specific contexts (such as, for instance, by journalists or academics). In recent years, the collective efforts of such platforms have become highly effective, with almost all terrorist content being removed before it has even been reported.
However, not all terrorist content is created equal. The removal of certain material needs to be prioritised. Problems of automation can arise, particularly in cases where technology is used to identify harmful content, which is then put before human reviewers to make a final decision; such a process can present a serious challenge around questions of what is prioritised for review, and how and when it is removed. Put another way, can technology be developed to assess the material effectively and accurately? A distinction needs to be drawn between materials that need immediate review and materials that can be placed in a queue. For example, one might consider the relative risk of a photograph showing a somewhat benign image of terrorist socialisation compared to a video depicting graphic violence.
Moreover, where automation has played a role, it has traditionally been deployed against the context in which social media posts exist rather than in relation to the content. As a result, these enquiries do not adequately harness the sorts of tools that are most practicable for tech company moderators.
This project explores how tech companies can draw that distinction in a timely and accurate fashion. Using IS as an initial test‑case, it will add nuance to how harmful content is classified. Our basic premise is that it is possible to codify the intent of harmful content by studying and then testing the logic behind its production. Indeed, if intent can be identified – that is, if a clear distinction can be drawn between tactical, action‑based content and strategic, brand‑based content – then it will be possible to better prioritise review and removal according to risk posed.
To this end, we aim to synthesise subject‑matter expertise with data science in this paper, using experimental text processing to interrogate and categorise our repository of official IS content. Our principal objective is to develop automated methods that can, when applied to similar bodies of materials (including those that are much larger), accelerate the process by which they may be disaggregated and, where relevant, triaged for moderation and/or referral. This will, in turn, help to enhance existing content moderation policies.
Given the overwhelming volume of content produced minute by minute, the need for such an approach is clearly pressing. On YouTube alone more than 300 hours of video are uploaded to the platform every minute, with users watching over a billion hours of video every day. There are, on average, 500 million tweets produced per day, totalling around 200 billion per year. During the first quarter of 2020, Facebook registered more than 2.6 billion monthly active users, while Instagram, which is owned by Facebook, hosts more than 500 million ‘Instagram Stories’ every day. The overwhelming majority of users on these platforms are, of course, there for wholly benign and legitimate purposes. There is no suggestion that this content needs to be censored or monitored in any way. However, the presence of malevolent violent extremists operating within the midst of this tsunami of material demonstrates the need for effective automated methods that are able to identify, parse and disaggregate the range of content that they produce.
This is why we have chosen to focus on IS material, which brought the issue of violent extremist content into sharp relief. On the whole, the broader jihadist movement has harnessed the use of technology for propaganda purposes better than other movements. In the 1990s, static websites, such as Azzam.com, brought news of jihadist campaigns in Chechnya, Bosnia and Afghanistan to English‑speaking audiences. Following the 2003 invasion of Iraq, password‑protected chat forums, such as Ansar al‑Mujahideen (‘supporters of the mujahideen’), Faloja (a reference to the Iraqi city of Fallujah, which became a hotbed of insurgent activity) and Shamukh (‘lofty’ or ‘someone to be looked up to’), became the primary forms of dissemination for violent extremist content, including videos and communiqués from groups like al‑Qaeda, al‑Shabaab and Boko Haram.
These forums were relatively static and insular environments. Thus violent extremist content had to be deliberately sought out as it existed in somewhat harder to reach corners of the internet. By the time of the Arab uprisings in 2011, social media had become the dominant form by which violent actors sought both to disseminate content and to win new recruits. The most dramatic encapsulation of this came with the rise of IS and the al‑Qaeda aligned Jabhat al‑Nusra between 2011 and 2016. The problem was not just their presence on these platforms but the fact that extremist content was now so easily available to anyone who wanted it – with a number of people encountering it purely accidentally. Consequently, the issue of how to address this became particularly acute for tech companies, law enforcement and counter‑terrorism policymakers.
Our paper aims to address this by exploring the ways in which automation might help with the identification and disaggregation of this content, allowing tech companies to identify such content more easily when it sits alongside material posted by legitimate users of their platforms.
Before proceeding, it is worth noting that the tools we have developed here have potential implications for other, non‑jihadist types of extremist content too, which individual companies will want to monitor based on their own needs. Harmful content online can take many forms, including abuse, bullying, harassment, threats, misogyny, hate speech and terrorist or violent propaganda, among others. These online activities can manifest themselves in real‑world harms, although perhaps none so dramatically as that relating to jihadist violence. In any case, they too require the kind of nuanced, contextualised moderation of which we speak below.
Read full report View infographic