Manara - Qatar Research Repository
10.1016_j.dim.2022.100019.pdf (3.2 MB)

PROVOKE: Toxicity trigger detection in conversations from the top 100 subreddits

Download (3.2 MB)
journal contribution
submitted on 2024-04-22, 08:56 and posted on 2024-04-22, 08:57 authored by Hind Almerekhi, Haewoon Kwak, Joni Salminen, Bernard J. Jansen

Promoting healthy discourse on community-based online platforms like Reddit can be challenging, especially when conversations show ominous signs of toxicity. Therefore, in this study, we find the turning points (i.e., toxicity triggers) making conversations toxic. Before finding toxicity triggers, we built and evaluated various machine learning models to detect toxicity from Reddit comments. Subsequently, we used our best-performing model, a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model that achieved an area under the receiver operating characteristic curve (AUC) score of 0.983 to detect toxicity. Next, we constructed conversation threads and used the toxicity prediction results to build a training set for detecting toxicity triggers. This procedure entailed using our large-scale dataset to refine toxicity triggers' definition and build a trigger detection dataset using 991,806 conversation threads from the top 100 communities on Reddit. Then, we extracted a set of sentiment shift, topical shift, and context-based features from the trigger detection dataset, using them to build a dual embedding biLSTM neural network that achieved an AUC score of 0.789. Our trigger detection dataset analysis showed that specific triggering keywords are common across all communities, like ‘racist’ and ‘women’. In contrast, other triggering keywords are specific to certain communities, like ‘overwatch’ in r/Games. Implications are that toxicity trigger detection algorithms can leverage generic approaches but must also tailor detections to specific communities.

Other Information

Published in: Data and Information Management
See article on publisher's website:



  • English



Publication Year

  • 2022

License statement

This Item is licensed under the Creative Commons Attribution 4.0 International License.

Institution affiliated with

  • Hamad Bin Khalifa University
  • College of Science and Engineering - HBKU
  • Qatar Computing Research Institute - HBKU

Usage metrics

    College of Science and Engineering - HBKU



    Ref. manager