Continuous Abstractive Recall-Oriented Understudy for Gisting Evaluation

Abstract

Summarization of scientific papers is a unique way of text summarization which allows us to use abstracts as human created labels for our dataset as they are written by the authors. This enables us to train our models using the collected data. In this work, we overview different approaches to text summarization and compare their results with different evaluation metrics. We demonstrate the downsides of ROUGE, the most commonly used summarization metric and introduce our own metric called CAROUGE, which gives more accurate scores for abstractive summaries. We also present our new dataset of 2000 scientific papers collected from arXiv. All experiments, described in this paper are performed on the data from our dataset, except for the final stage of our project where we involve 5 human judges to do a manual summarization of several papers from the dataset.