RoMe: A Robust Metric for Evaluating Natural Language Generation
Abstract: Evaluating Natural Language Generation (NLG) systems is a challenging task. Firstly, the metric should ensure that the generated hypothesis reflects the referenceâs semantics. Secondly, it should consider the grammatical quality of the generated sentence. Thirdly, it should be robust enough to handle various surface forms of the generated sentence. Thus, an effective evaluation metric has to be multifaceted. In this paper, we propose an automatic evaluation metric incorporating several core aspects of natural language understanding (language competence, syntactic and semantic variation). Our proposed metric, RoMe, is trained on language features such as semantic similarity combined with tree edit distance and grammatical acceptability, using a self-supervised neural network to assess the overall quality of the generated sentence. Moreover, we perform an extensive robustness analysis of the state-of-the-art methods and RoMe. Empirical results suggest that RoMe has a stronger correlation to human judgment over state-of-the-art metrics in evaluating system-generated sentences across several NLG tasks.
Show BibTeX
@inproceedings{DBLP:conf/acl/RonyKCU022,
author = {Md. Rashad Al Hasan Rony and
Liubov Kovriguina and
Debanjan Chaudhuri and
Ricardo Usbeck and
Jens Lehmann},
editor = {Smaranda Muresan and
Preslav Nakov and
Aline Villavicencio},
title = {RoMe: {A} Robust Metric for Evaluating Natural Language Generation},
booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), {ACL} 2022, Dublin, Ireland,
May 22-27, 2022},
pages = {5645--5657},
publisher = {Association for Computational Linguistics},
year = {2022},
url = {https://doi.org/10.18653/v1/2022.acl-long.387},
doi = {10.18653/V1/2022.ACL-LONG.387},
timestamp = {Mon, 01 Aug 2022 16:27:50 +0200},
biburl = {https://dblp.org/rec/conf/acl/RonyKCU022.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}