This Insight was published as part of GIFCT’s Working Group on Gaming and Youth (GYWG). GIFCT Working Groups bring together experts from diverse stakeholder groups, geographies and disciplines to offer advice in specific thematic areas and deliver on targeted, substantive projects.
Gaming environments have become some of the most significant social spaces in contemporary online life. Multiplayer games, livestreaming services, forums, and gaming-adjacent spaces – services and platforms such as Discord that are used to socialise around games – provide environments where friendships form, social status is negotiated, and communities develop shared cultures and norms.
Existing approaches to extremist activity online have focused on content moderation, detection, and removal. Representative examples include Riot Games’ handling of reported disruptive behaviour, AI-assisted voice detection in Call of Duty, and Discord’s documented removal of accounts and servers for violent extremism. These address different harms but illustrate enforcement through content moderation tools. Enforcement remains essential; it cannot, on its own, address influence that develops through relationships, humour and shifting social norms. Extremist engagement rarely begins with explicit ideological recruitment; pathways may instead emerge through friendship networks, shared grievances, or a search for belonging and recognition.
Responding effectively, therefore, requires moving beyond a reactive “ban hammer” model. Prevention must address the social processes through which influence develops. Building on participation in GIFCT’s Gaming and Youth Radicalisation Work Group and our previous research on extremist identity formation and influence among young users on gaming-adjacent platforms, this Insight proposes a four-part framework: education, friction, redirection, and intervention. It treats gaming environments not simply as spaces of risk, but as communities where users can develop resilience.
Context: Gaming Environments as Spaces of Identity Formation
Understanding extremist exploitation within gaming environments requires first recognising the central role these spaces play in identity development. Users are not passive consumers of digital content; they are active participants who develop relationships, perform identities, and create meaning through interaction with others.
Online multiplayer games are particularly powerful identity-forming spaces because they rely heavily on cooperation and shared experience. Team-based gameplay encourages users to work together, develop trust, and build collective identities around shared goals. Kowert and Oldmeadow show that online gaming communities can create meaningful social connections, with group membership becoming an important part of how individuals understand themselves and their relationships with others. One user’s gaming identity may span across platforms, with in-game interaction, Discord servers, livestreams, and wider online communities representing interconnected social spaces and opportunities for interaction with others. These dynamics are especially significant for younger users. Adolescence and early adulthood involve identity exploration and heightened sensitivity to peer influence. Young players use gaming spaces to explore communities, values, humour, and social identities. This is not inherently problematic; it simply reflects youth development increasingly occurring online.
At the same time, extremists and other bad actors have adapted to these decentralised environments through loose networks rather than formal organisations, and have spread their positions through memes, aesthetics, cultural references, and informal communities. Gaming-adjacent spaces may become attractive to young people not because they are inherently extremist, but because they offer opportunities for participation, social bonding, and identity-building. Research finds that extremist engagement can move effectively across digital ecosystems, with users adapting identities to different platform norms and social contexts. And while these social spaces and processes are not inherently extremist, valid concerns are raised when these can be manipulated by extremists with harmful intentions.
A Four-Part Prevention Framework for Gaming and Gaming-Adjacent Spaces
1. Education: Building Resilience Before Exposure
Preventing extremist exploitation should begin with a rethink of how young users learn community standards, acceptable behaviour, and online manipulation. Written community guidelines and acceptable-use policies remain necessary, but often fail to reflect how young people learn norms in gaming environments. Players learn through participation: observing experienced users and creators, adapting to peers, and seeing what communities reward or discourage. Short, interactive scenarios can let users practise recognising pressure and asking for help before they encounter it in a live conversation.
Therefore, one of the most effective ways to educate young users is to move from rule awareness to a form of literacy solely based on identifying harms. Community guidelines should not simply explain what behaviour will result in punishment; they should help players understand why certain behaviours threaten the safety and integrity of the spaces they value.
A useful example is the increasing use of interactive onboarding and digital citizenship models by gaming-adjacent platforms, where users learn through realistic scenarios rather than passive reading. Platforms could introduce short “safety missions” or decision-based tutorials similar to existing gameplay mechanics. For example, a young player might encounter a scenario where a new teammate offers in-game rewards in exchange for personal information, pressures them to join a private server, or encourages increasingly hostile jokes. Instead of simply labelling these behaviours as prohibited, such educational interventions and modules could explain the underlying manipulation tactics, such as trust-building, isolation, or boundary testing. This approach reflects how gaming environments already teach users complex systems. Players learn the mechanics of play through teamwork and group strategy; safety behaviours can be developed in the same way.
Existing approaches provide useful foundations (see Appendix A). Roblox’s Safety and Civility programmes combine moderation, reporting, parental controls, and age-sensitive educational material. Common Sense Education’s digital citizenship curriculum and Internet Matters’ Digital Matters programme use scenario-based learning on relationships, privacy, and harmful interactions. Although not exclusively focused on extremism, they show how prevention can be embedded in broader digital resilience-building.
Extremist grooming and exploitation can begin through friendship, humour, belonging, recognition, and shared identity, not just violative content. The 764 network illustrates how apparent friendship can become a route into coercive exploitation. In one documented UK case, a 764-linked offender contacted his victim through Roblox before moving the conversation to Discord, where initially minor requests escalated into blackmail and demands for self-harm. Education should therefore build young users’ literacy in manipulation, so they can understand why someone might move a conversation off a moderated platform or encourage harmful behaviour. This is particularly important in gaming environments where identity formation is closely tied to teams, clans, guilds, and online communities, and prevention should reinforce positive gamer identities instead of portraying young people only as potential victims.
2. Friction: Introducing Safety at Moments of Risk
Education builds resilience before exposure, but platforms also need to integrate support during risky interactions. Friction-based interventions are small design features that create opportunities for reflection without immediately restricting autonomy. Exploitation often depends on momentum: rapid trust-building, emotional dependence, movement into private spaces, and gradual shifts in acceptable behaviour. Friction at critical moments can interrupt these processes.
Discord’s safety alerts provide a player-facing example in a gaming-adjacent service. When Discord identifies a need for an alert on a first-time sender’s direct message to a teen, it encourages the recipient to reconsider replying. The recipient can open safety tips or block the sender; the tips also offer a mute option. Alerts are not shown for every new sender. The practical intervention is a pause inside the conversation, with protective actions immediately available.
Platforms could adapt this principle when a new contact repeatedly pressures a young player to move to a less moderated space. A contextual notice could explain the change in protections and offer options to remain, decline or seek help. This is a proposed design, not a quotation from an existing interface. Moving between platforms is routine gaming behaviour; pressure, secrecy and repeated attempts at coercion should inform the response, rather than an invitation alone.
Behavioural prompts are already used in broader trust-and-safety contexts to encourage users to reconsider potentially harmful posts before publication, as such measures shift intervention from post-event punishment to prevention before escalation. As one example, Microsoft and Xbox Community Standards combine behavioural expectations with reputation systems, reporting mechanisms, transparency around enforcement, and reminders about positive participation. These approaches communicate that community membership involves responsibility.
Similarly, Riot Games’ player behaviour systems use behavioural moderation warnings, feedback, and consequences to encourage players to adjust harmful communication before problems escalate. Rather than relying only on punishment, these systems attempt to reinforce community norms. The League of Legends Honor system represents another example of positive behavioural design. By rewarding cooperation, teamwork, and respectful behaviour, the system encourages users to associate social status with positive community participation rather than only competitive success.
3. Redirection: Replacing Harmful Pathways with Positive Communities
A limitation of enforcement-focused moderation is that removing harmful content does not address why users entered those spaces. They may initially be seeking belonging, humour, friendship, recognition, or status rather than ideology. In gaming ecosystems, where community membership can be central to identity, removal without alternatives can leave those social needs unresolved (Argaman, 2026). Users may therefore return to harmful communities or follow the same contacts into less moderated spaces, where extremist actors can continue to exploit their need for belonging with fewer opportunities for detection and support.
Innovative redirection strategies focus on guiding users toward healthier spaces that provide similar benefits. For example, rather than simply removing a user from a toxic gaming community, platforms could recommend positive communities or other positive interventions built around shared interests, competitive play, creativity, or mentorship. Existing counter-extremism initiatives such as Moonshot and Jigsaw’s Redirect Method show the value of offering alternative pathways at moments of curiosity or vulnerability. In gaming environments, similar approaches could recommend positive creators, supportive communities, e-sports opportunities, or educational content that meets users’ needs for identity and belonging.
Counter-speech initiatives such as Project One-to-One and Counter Conversations, developed by organisations including the Institute for Strategic Dialogue (ISD), similarly challenge extremist narratives and provide alternative perspectives. They recognise that prevention requires not only removal but meaningful alternatives and a human element.
For a player, a proposed gaming alternative could appear after they leave or report an abusive group. Alongside confirmation of their action, the platform could offer an optional route to a moderated team-finding space for the same game, a supervised tournament or a creative community. A useful recommendation would explain the community’s age suitability, moderation arrangements and joining process. This gives the player a concrete way to continue the activity and social connection they value.
Credibility matters more than merely providing another link. Platforms should develop alternatives with players, trusted creators and community organisations, ensuring there is moderation capacity to receive newcomers. Recommendations should be voluntary and relevant to the player’s interests, without labelling someone as an extremist because they encountered harmful material. Evaluation should examine whether users find sustained, safer participation, rather than counting clicks alone.
4. Intervention: Responding Before Harm Escalates
The final component involves identifying and responding to users who may already be moving along harmful pathways. In our research on gaming-adjacent spaces, practitioners describe attention to relationships and dependency as well as explicit content. This supports examining patterns, such as repeated isolation attempts combined with threats or demands to target others. Dr Heath Landress’s work on nihilistic aesthetics offers a complementary perspective on how visual cues might inform assessments of escalation. Such cues should prompt contextual assessment rather than serve as standalone evidence: a private friendship, an unfamiliar joke or a change in group membership is not sufficient evidence of extremist exploitation.
Grooming and extremist exploitation are often better understood as behavioural processes.
Effective intervention should therefore focus on patterns of escalation. These might include sudden movement into increasingly closed communities, repeated attempts by others to isolate a user, rapid adoption of harmful identity markers, or escalating engagement with extreme material. Intervention can then combine human moderation with technology-assisted detection to identify signals such as coercive communication, coordinated harassment, or attempts to move users into less visible environments. Human judgement remains essential because gaming cultures are highly contextual and norms around humour and language vary widely.
Modulate’s ToxMod, for example, uses AI-assisted voice moderation to identify harmful communication patterns, including severe harassment and threatening behaviour within multiplayer environments. While automated systems cannot replace human judgement, they can support moderators managing large-scale communities. This shows how detection becomes visible to the player. It is a general conduct-enforcement example, not a demonstrated system for identifying extremist grooming.
Where coercion or exploitation is suspected, trained moderators should assess context and choose a proportionate response. This could combine restrictions on the person causing harm with support for the affected user and referral to appropriate safeguarding specialists. Human judgement, clear reasons and routes to challenge mistakes are essential: gaming language varies, and automated signals cannot establish intent on their own.
Platforms such as Discord have also developed crisis intervention approaches combining reporting systems, moderation, safety resources, and escalation routes for serious risks. These approaches recognise that some situations require safeguarding responses rather than standard enforcement actions.
Bystanders also need concrete, manageable actions. ADL’s Online Hate and Harassment Safety Guide links users to platform-specific reporting and safety tools. Building on that practical approach, communities should encourage players to report the relevant message or incident, use mute or block controls, and check privately whether the targeted teammate wants support. Children should know how to contact a trusted adult. Players should not be expected to investigate suspected recruiters or confront them publicly; responsibility for assessment rests with trained teams.
Conclusion: From Risk Management to Resilience Building
Gaming environments should not be understood solely through the lens of threat. They are important social spaces where millions of people build relationships, develop skills, and construct identities. The same mechanisms that can be exploited by harmful actors – belonging, recognition, influence, and community – can also become the foundation for prevention. Future approaches to countering extremist exploitation in gaming spaces should therefore move beyond asking only how harmful content can be removed. A more effective question is how platforms, communities, and practitioners can create environments that are more resistant to manipulation and coercion.
A model based on education, friction, redirection, and intervention recognises that safety is not achieved simply through restriction. It comes from helping young users understand influence, supporting better decisions, providing positive communities, and responding before harm escalates. Safer gaming depends on strengthening what makes games meaningful – identity, belonging, creativity, and connection – as protective factors against extremist exploitation and other online harms.
—
Dr William Allchorn is a Senior Research Fellow at the International Policing and Public Policing Research Institute (IPPPRI), Anglia Ruskin University, and an expert on radical-right extremist social movements in the UK, Europe, and beyond. He has recently advised the UK, US and Australian governments on their approaches to countering the extreme right-wing, and has undertaken some of the first metrical testing of violent far-right extremist counter-narratives in the UK, US and Australia with governments (e.g. EU Internet Forum), civil society organisations (e.g. Counter-Extremism Project), and tech companies (e.g. Meta) across the world.
Dr Elisa Orofino is a Senior Research Fellow and Academic Lead for the Extremism and Counter-Terrorism Sub-Theme at the International Policing and Public Policing Research Institute (IPPPRI), Anglia Ruskin University, and an expert on Islamist extremist social movements in the UK, Europe, and beyond.
—
Appendix A: Comprehensive List of Interventions (Segmented by Age Group)
| Intervention | Example | How it works | Main audience |
| Internet Matters ’Digital Matters’ | Scenario-based learning where young users make choices about online situations and see consequences | Children 9–14 |
| Common Sense Education | Lessons on privacy, manipulation, online relationships, misinformation, hate speech, and respectful communities | Children & teens |
| Roblox Safety & Civility programmes | Age-appropriate rules, reporting education, parental controls, moderation explanations, safety resources | Younger players |
| Microsoft / Xbox Community Standards | Uses reputation systems, reporting tools, enforcement transparency, and reminders of expected behaviour | All players |
| Riot Games player behaviour systems | Gives warnings, feedback, and penalties for harmful communication; encourages norm correction before escalation | Teens & adults |
| League of Legends Honor system | Rewards cooperative behaviour and positive teamwork rather than only punishing misconduct | Teens |
| Modulate ToxMod | Detects harmful voice-chat patterns including harassment, threats, and severe misconduct | Multiplayer communities |
| Discord safety interventions | Combines reporting, moderation, safety resources, and escalation routes for high-risk situations | Teens & young adults |
| Institute for Strategic Dialogue programmes | Builds resilience by exposing manipulation tactics and promoting alternative narratives | Teens & young adults |
| Moonshot Redirect Method | Uses online signals to redirect people searching harmful content toward constructive resources | Older teens/adults |
| Anti-Defamation League gaming community resources | Trains players to recognise hate, harassment, and intervene safely | Teens/adults |
| National Center for Missing & Exploited Children NetSmartz | Teaches young people how to identify grooming, unsafe contact, and report concerns | Children & teens |
—
Are you a tech company interested in strengthening your capacity to counter terrorist and violent extremist activity online? Apply for GIFCT membership to join over 30 other tech platforms working together to prevent terrorists and violent extremists from exploiting online platforms by leveraging technology, expertise, and cross-sector partnerships.
The views and opinions expressed in this Insight are the authors’ own and do not necessarily reflect those of GNET or any of its partners.
