Introduction
Artificial intelligence (AI) has become integrated in every field—from infrastructure to medicine to law enforcement and education—and is also playing a role in countering violent extremism (CVE) online. AI provides diverse tools for analysing, detecting, and disrupting extremist activities, but its use in this field presents not only advantages but also challenges for practitioners.
This Insight discusses how extremists leverage AI to enhance their presence within communities online. It also discusses how AI technologies can be used to counter these risks effectively and delves into the ethical and practical obstacles presented by AI applications and their impact on government policies and strategies. As the threat from extremist actors harnessing AI for malign purposes increases, the strategies and technologies employed to counter it should increase as well.
Examining AI’s Role in CVE
1 – AI-Powered Content Moderation
AI tools like Natural Language Processing (NLP) models, such as BERT (Bidirectional Encoder Representations from Transformers), have become an essential tool in identifying hate speech and extremist rhetoric. Unlike word-embedding machine learning, which learns word meanings based only on the preceding words, BERT’s two-way approach allows it to consider both the previous and following context, leading to a better understanding of the language. This technology is particularly useful for sifting through data to pinpoint patterns associated with radicalisation. In a recent analysis of far-right Telegram channels, NLP tools detected coded language used to escape detection. For example, these channels employ words like “defend” and “jews” in antisemitic contexts and terms like “white genocide” to promote racist ideologies. The use of terms with similar meanings was found to have deep-rooted ideological significance, allowing researchers to identify extremist ideologies across the platforms.
The model BERT can also help detect and classify content in multiple languages. For example, platforms like Hatebase support NLP models to create databases of hate speech that help identify and track extremist trends globally. This ability to work across languages is particularly handy for identifying far-right narratives shared in regions where less commonly spoken languages might otherwise avoid moderation. The problem of detecting sarcasm and nuanced language can also be overcome with the new developments of NLP models. The NLP algorithms can identify certain linguistic phrases that often include sarcasm, such as exaggerated praise or the use of irony. For instance, Al-Qaeda uses sarcastic humour to ridicule adversaries, as shown in an example of the group mocking President Donald Trump: “Trump and many of his aides vehemently opposed the military option. They prefer to rely on economic sanctions, besides of course ‘Presidential’ tweets, which might force Tehran to bend to American demands or, after another forty years, somehow bring about the collapse of the regime!”. By analysing syntactic structures and semantic relationships within a sentence, NLP models can help uncover these strategic uses of sarcasm in extremist discourse en masse.
2 – Detection of Extremist Visual Content
Real-time object detection models, such as YOLO (You Only Look Once), could provide valuable capabilities for identifying extremist symbols and imagery. This technology could be highly useful in scanning through multimedia content shared across social media platforms such as Telegram or extremist online forums. While there are currently no examples of using YOLO to identify extremist content online, YOLO has been successfully used in various scenarios that demand quick recognition of particular objects. The tool was created to spot shapes and sizes in old maps and papers, and also to detect forbidden objects in X-ray scans—a demonstration of its potential usefulness in security scenarios where quick identification is crucial.
Training YOLO to recognise propaganda content with prohibited symbols could help identify dangerous online material and enable law enforcement to take swift action to curb the spread of extremist content. Furthermore, YOLO could assist in law enforcement investigations by checking seized devices and mapping links between people through materials or symbols they have in common. This application of YOLO can help in understanding the structures and ranks within organisations. However, to use the app effectively against extremism, a comprehensive dataset of prohibited symbols and materials would be needed to ensure proper use and detection.
3 – Tackling Manipulated Media
Advanced AI tools for identifying deepfakes are being developed to combat the rise of false information and the exploitation of communities susceptible to extremist agendas. Recent cases showed the use of deepfake videos to misattribute statements to political figures, increasing tensions in already unstable political environments. For example, during the 2024 US elections, deepfake detection software uncovered manipulated videos of Vice President Kamala Harris. The fake audio includes comments about diversity, President Joe Biden and border policies, as well as edited clips from real Kamara Harris appearances. Additionally, in March 2022, a deepfake video showed Ukrainian President Volodymyr Zelenskyy telling Ukrainian soldiers to put down their weapons and surrender to the Russians. Deepfake detection tools can identify these manipulations, preventing further escalation in fraught environments. One example is a new tool created by Meta called AudioSeal. This tool can help tackle the increasing use of voice cloning technology for scams, misinformation, and exploitation. AudioSeal differs from traditional methods because its localised detection approach makes it faster and more efficient than complex models.
A major challenge with deepfake detection tools is the rapid evolution of deepfake technology used by extremists. They may leverage this technology to amplify their influence online as a recruitment tool. This highlights the necessity of finding better solutions to ensure the authenticity of the content or better-updated detection systems. Furthermore, as these detection systems are easily available for purchase, concerns about their exploitation increase, not only for extremist activities but also for political campaigns.
Ethical and Operational Considerations
1 – Algorithmic Bias
AI systems are not resistant to the biases inserted in their training data. For example, a tool designed to detect extremist content online might target regular users rather than extremist ones, potentially weakening trust in AI solutions. Since most major tech companies are located in the West (and China), the algorithm might be unfamiliar with certain types of violence and discrimination due to cultural differences.
Research has shown that automated moderation tools on social platforms have targeted content from marginalised communities, raising concerns about equitable treatment. When people share their perspectives and racialised experiences online, content moderation algorithms may find difficulties in distinguishing between race-related talk and racist talk. For these reasons, human reviewers may opt to remove race-related content, declaring such content uncomfortable, inappropriate, or contentious.
Recently, for example, posts about the LGBTQIA+ community on Facebook, such as those promoting a suicide prevention support service called QLIFE, were deleted. Meta has described the situation as a “technical error”, however, the individuals and communities were affected nonetheless. Accidental or intentional moderation bias often arises from the overrepresentation of certain terms or behaviours in training datasets that lack proper contextualisation during AI development. To mitigate these biases, systematic security checks of AI algorithms are essential. Additionally, partnerships with civil society organisations can ensure a more comprehensive and fair application of AI.
2 – Privacy Concerns
The deployment of AI often requires a huge amount of data collection, raising concerns about privacy, civil liberties and freedom of expression. Policymakers face the ongoing challenge of balancing security needs with individual rights, particularly when addressing extremist content online. The UN has identified the “unlawful or arbitrary collection of personal data” as a privacy violation that undermines democratic principles. AI algorithms can also amplify biases in their training data, increasing risks of discrimination. CVE tools, such as those reliant on automated profiling and mass data collection, frequently intrude on individual privacy, can deteriorate trust between citizens and governments, and potentially foster “digital authoritarian regimes.”
While automated processes may reduce human biases, critics argue that real-time surveillance of social media and online platforms is overly intrusive and often incompatible with human rights. Contrarily, if used responsibly, AI tools that examine lawfully obtained data to identify patterns or match suspects may comply with privacy standards. A constant challenge is to achieve a balance between protecting human rights and effective security measures.
3 – Psychological Harm and Societal Impact
The deployment of AI tools in counter-extremism initiatives can have unintended psychological impacts on non-extremist individuals and communities. For instance, banning some posts due to false positives in content moderation can contribute to mistrust and indignation from people, as they feel attacked without cause. Furthermore, the exposure of extremist content online, even if flagged, can still harm specific vulnerable audiences. Research has shown that exposure to hateful content can amplify feelings of alienation, particularly among young individuals who may already be vulnerable to radicalisation. For survivors of extremist violence, encountering this content online can serve as a re-traumatising experience.
4 – Evolving Extremist Tactics in AI Deployment
The implementation of AI tools in counter-extremism initiatives presents dual-use risks, as authoritarian regimes can misuse the same technologies to target other states or marginalised groups. Russian use of digital restrictions, as well as AI-based manipulation and election interference methods, have been widely debated since the US elections in 2016, and it can be seen interfering even more in the elections of various EU countries in 2024. Adopting international regulations is important to reduce these risks, even if many Western companies prefer to engage with authoritarian regimes due to the sheer scale of potential markets being presented by these countries.
Additionally, extremist groups are adept at exploiting AI’s vulnerabilities, including jailbreaking generative AI systems to create propaganda. For example, researchers identified cases in the US, UK, and elsewhere where extremists used AI tools for propaganda and disinformation campaigns online, indicating the up-to-date measures used by extremists to exploit technological gaps. These applications underline the necessity of being one step ahead by implementing international safeguards and continuous adaptation in the use of AI for counter-extremism efforts.
Conclusion and Policy Recommendations
To conclude, recommendations that should be explored to combat online extremism include introducing a comprehensive strategy to increase detection capabilities online, transparency and accountability in the process, better international collaboration among the actors, education, and in-depth research. Sophisticated AI models must be developed to improve coded language detection and visual content detection, supported by red teams anticipating and countering extremist evasion tactics.
Collaboration among tech companies is essential, with shared databases of extremist symbols and narratives enabling better AI training. Transparency and accountability should be prioritised through mandatory checks to identify biases and the establishment of independent oversight bodies to ensure ethical use. Other aspects are public awareness campaigns and introducing digital literacy into school schedules, which can empower users to recognise and counter manipulated content. Last but not least, investment in research and development is paramount to strengthening AI capabilities in CVE efforts while protecting human rights.
The use of AI has the potential to counter violent extremism, providing a diversity of tools to disrupt harmful activities online. However, these instruments also bring challenges that require careful navigation and application of some of the recommendations mentioned above. Increasing AI’s role in CVE will require sustained innovation, ethical oversight, and multistakeholder cooperation to address the evolving threat landscape effectively.
Clarisa Nelu is a research analyst in the Observatory on Human Rights at CeSPI, where she examines the intersection of human rights and migration with emerging technologies, particularly artificial intelligence. Previously, she was an intern/research analyst in the Research Division of the NATO Defence College, focusing on challenges in the Black Sea region and the strategic implications of AI for security and defence. She also completed an internship at the International Center for Counter-Terrorism, where she investigated the use of generative AI by terrorist groups. Her broader research interests span global security challenges, including terrorism, human trafficking, migration, human rights violations, and the transformative impact of new technologies on these critical issues.