Overview of RefutES at IberLEF 2024Automatic Generation of Counter Speech in Spanish

  1. Vallecillo-Rodríguez, María Estrella
  2. Cantero-Romero, María Victoria
  3. Cabrera-de-Castro, Isabel
  4. Ureña-López, Luis Alfonso
  5. Montejo-Ráez, Arturo
  6. Martín-Valdivia, María Teresa
Journal:
Procesamiento del lenguaje natural

ISSN: 1135-5948

Year of publication: 2024

Issue: 73

Pages: 449-459

Type: Article

DIALNET GOOGLE SCHOLAR lock_openOpen access editor

More publications in: Procesamiento del lenguaje natural

Abstract

This paper presents an overview of RefutES 2024, organized at IberLEF 2024 and co-located with the 40th International Conference of the Spanish Society for Natural Language Processing (SEPLN 2024). The main purpose of RefutES is to promote research on the automatic generation of counter speech in Spanish. Counter speech generation is a new strategy developed to combat hate speech on social media that involves generating a response that negates the offensive message. In this shared task, participants must be able to generate a response to hate speech messages directed at various targets of offense in Spanish. The response should be reasoned, respectful, non-offensive, and contain specific and truthful information. Moreover, we asked participants to submit measurements of carbon emissions for their systems, emphasizing the need for sustainable NLP practices. In this first edition, a total of 6 teams signed up to participate in the task, 1 submitted official runs on the test data, and 1 submitted system description papers.

Bibliographic References

  • Benesch, S. 2014. Countering dangerous speech: New ideas for genocide prevention. URL: https://ssrn.com/abstract=3686876.
  • Bengoetxea, J., Y.-L. Chung, M. Guerini, and R. Agerri. 2024. Basque and Spanish counter narrative generation: Data creation and evaluation. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2132–2141, Torino, Italia, May. ELRA and ICCL. URL: https://aclanthology.org/2024.lrecmain. 192.
  • Bonaldi, H., Y.-L. Chung, G. Abercrombie, and M. Guerini. 2024. Nlp for counter-speech against hate: A survey and how-to guide.
  • Chiruzzo, L., S. M. Jiménez-Zafra, and F. Rangel. 2024. Overview of IberLEF 2024: Natural Language Processing Challenges for Spanish and other Iberian Languages. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference of the Spanish Society for Natural Language Processing (SEPLN 2024), CEUR-WS.org.
  • Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. 2019. Unsupervised crosslingual representation learning at scale. CoRR, abs/1911.02116.
  • Courty, B., V. Schmidt, S. Luccioni, Goyal-Kamal, MarionCoutarel, B. Feld, J. Lecourt, LiamConnell, A. Saboni, Inimaz, supatomic, M. Léval, L. Blanche, A. Cruveiller, ouminasara, F. Zhao, A. Joshi, A. Bogroff, H. de Lavoreille, N. Laskaris, E. Abati, D. Blank, Z. Wang, A. Catovic, M. Alencon, M. Stęchły, C. Bauer, Lucas-Otavio, JPW, and MinervaBooks. 2024. mlco2/codecarbon: v2.4.1, May.
  • Dettmers, T., A. Pagnoni, A. Holtzman, and L. Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs, May. arXiv:2305.14314 [cs].
  • Fanton, M., H. Bonaldi, S. S. Tekiroğ lu, and M. Guerini. 2021. Human-in-the-loop for data collection: a multi-target counter narrative dataset to fight online hate speech. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics.
  • Furman, D., P. Torres, J. Rodríguez, D. Letzen, M. Martinez, and L. Alemany. 2023. High-quality argumentative information in low resources approaches improve counter-narrative generation. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2942–2956, Singapore, December. Association for Computational Linguistics.
  • Halim, S. M., S. Irtiza, Y. Hu, L. Khan, and B. Thuraisingham. 2023. Wokegpt: Improving counterspeech generation against online hate speech by intelligently augmenting datasets using a novel metric. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1–10. IEEE.
  • Mathew, B., H. Tharad, S. Rajgaria, P. Singhania, S. K. Maity, P. Goyal, and A. Mukherjee. 2018. Thou shalt not hate: Countering online hate speech. In International Conference on Web and Social Media.
  • Qian, J., A. Bethke, Y. Liu, E. Belding, and W. Y. Wang. 2019. A benchmark dataset for learning to intervene in online hate speech. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pages 4755–4764, Hong Kong, China, November. Association for Computational Linguistics.
  • Schieb, C. and M. Preuss. 2016. Governing hate speech by means of counterspeech on facebook. In 66th ica annual conference, at fukuoka, japan, pages 1–23.
  • Tekiroglu, S., H. Bonaldi, M. Fanton, and M. Guerini. 2022. Using pre-trained language models for producing counter narratives against hate speech: a comparative study. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3099–3114.
  • Touvron, H., L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models, July. arXiv:2307.09288 [cs].
  • Vallecillo-Rodríguez, M.-E., M.-V. Cantero-Romero, I. Cabrera-De-Castro, A. Montejo-Ráez, and M.-T. Martín-Valdivia. 2024. CONAN-MT-SP: A Spanish corpus for counternarrative using GPT models. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3677–3688, Torino, Italy, May. ELRA and ICCL.
  • Vallecillo-Rodríguez, M. E., A. Montejo-Raéz, and M. T. Martín-Valdivia. 2023. Automatic counter-narrative generation for hate speech in spanish. Procesamiento del Lenguaje Natural, 71(0):227–245.
  • Zhang, T., V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. 2019. BERTScore: Evaluating Text Generation with BERT. September.
  • Zhao, W., M. Peyrard, F. Liu, Y. Gao, C. M. Meyer, and S. Eger. 2019. Mover-Score: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance. In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563–578, Hong Kong, China, November. Association for Computational Linguistics.
  • Zubiaga, I., A. Soroa, and R. Agerri. 2024. Ixa at refutes 2024: Leveraging language models for counter narrative generation. In IberLEF (Working Notes). CEUR Workshop Proceedings.
Data source: Dialnet