AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages - Cnam - Conservatoire national des arts et métiers
Communication Dans Un Congrès Année : 2024

AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

1 UCL - University College of London [London]
2 Masakhane NLP
3 University of Maryland [Baltimore]
4 Unbabel
5 IST - Instituto Superior Técnico, Universidade Técnica de Lisboa
6 INESC-ID - Instituto de Engenharia de Sistemas e Computadores Investigação e Desenvolvimento em Lisboa
7 ENSIAS - Ecole Nationale Supérieure d'Informatique et d'Analyses des Systèmes
8 SADiLaR - South African Centre for Digital Language Resources
9 Aston University [Birmingham]
10 University of Eastern Finland
11 LUT - Luleå University of Technology = Luleå Tekniska Universitet
12 Fudan University [Shanghai]
13 CEDRIC - VERTIGO - CEDRIC. Données complexes, apprentissage et représentations
14 Lelapa AI
15 Imperial College London
16 HausaNLP - Hausa natural language processing
17 DEUSTO - Universidad de Deusto
18 UC - University of California
19 Lancaster University
20 UM5 - Université Mohammed V de Rabat [Agdal]
21 JUST - Jamhuriya University Of Science and Technology
22 LAUTECH - Ladoke Akintola University of Technology
23 The College of Saint Rose
24 UMN - University of Minnesota [Twin Cities]
25 Microsoft Research
26 UvA - University of Amsterdam [Amsterdam] = Universiteit van Amsterdam
27 TUK - Technical University of Kenya
28 AIMS - African Institute for Mathematical Sciences
29 KU Leuven - Catholic University of Leuven = Katholieke Universiteit Leuven
30 SIAT - Shenzhen Institute of Advanced Technology [Shenzhen]
31 KASU - Kaduna State University
32 University of Cape Coast [Ghana]
33 Ghana NLP - Ghana Natural Language Processing
34 KNUST - Kwame Nkrumah University of Science and Technology
35 New Mexico State University
36 USIU-Africa - United States International University-Africa
37 NAU-UNIZIK - Nnamdi Azikiwe University
Wangui Kimotho
  • Fonction : Auteur
Anuoluwapo Aremu
  • Fonction : Auteur
Jessica Ojo
  • Fonction : Auteur
Perez Ogayo
  • Fonction : Auteur
Oumaima Hourrane
  • Fonction : Auteur
Lolwethu Ndolela
  • Fonction : Auteur
Thabiso Mangwana
  • Fonction : Auteur
Eric Muchiri
  • Fonction : Auteur
Daud Abolade
  • Fonction : Auteur
Simbiat Ajao
  • Fonction : Auteur
Iyanuoluwa Shode
  • Fonction : Auteur
Ricky Macharm
  • Fonction : Auteur
Abeeb Afolabi
  • Fonction : Auteur
Nnaemeka Obiefuna
  • Fonction : Auteur
Onyekachi Ogbu
  • Fonction : Auteur
Sam Ochieng’
  • Fonction : Auteur

Résumé

Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AFRICOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441). © 2024 Association for Computational Linguistics English-Egyptian Arabic (eng-arz), English-French (eng-fra)—a control LP, English-Hausa (eng-hau), English-Igbo (eng-ibo), English-Kikuyu (eng-kik), English-Luo (eng-luo), English-Somali (eng-som), English-Swahili (eng-swh), English-Twi (eng-twi), English-isiXhosa (eng-xho), English-Yoruba (eng-yor), and Yoruba-English (yor-eng). Moreover, we extend our annotation collection to include domain-specific texts from News, TED talks, Movies, and IT domains for English-Yoruba translations, which were established in prior research by Adelani et al. (2021) and Shode et al. (2022), ensuring a comprehensive and domain-varied evaluation. We provide the information of language family groups that our targeted African languages belong to in Table 4 of Appendix A.1.
Fichier non déposé

Dates et versions

hal-04676542 , version 1 (23-08-2024)

Identifiants

Citer

Jiayi Wang, David Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, et al.. AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages. 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies, Association for Computational Linguistics, Jun 2024, Mexico City, Mexico. pp.5997-6023, ⟨10.18653/v1/2024.naacl-long.334⟩. ⟨hal-04676542⟩
22 Consultations
0 Téléchargements

Altmetric

Partager

More