DiFiLE: A Knowledge-Distillation Longformer Model for Finance with Ensembling

Abstract

10-K reports are a very important source of information in finance. Unfortunately, due to their text length they can hardly be analyzed by state-of-the-art transformer-based methods. In this paper, we aim to address this by combining the fields of efficient attention mechanisms, knowledge distillation (KD), and ensembling. Our five-step approach, DiFiLE, first pre-processes the data and splits it into data chunks based on the report items. Then, for each chunk, we estimate a teacher Longformer model. This is followed by KD and the generation of the corresponding student models. Finally, we aggregate the results from the chunks with ensembling and in particular stacking. We evaluate DiFiLE on the 10-K reports of the DJIA companies. The results show high performance of the teacher model, which is then well mimicked by its distilled version, requiring 30% fewer resources.