Evaluation of Information Extraction Algorithms for Preserving Analogical Semantics within Knowledge Graphs

Abstract

Analogical reasoning is a promising, lightweight solution to inferencing on novel data without prior training. However, algorithms with this methodology have historically relied on strict, human-defined schemas. To address this issue, this work proposes using information extraction algorithms to transform textual analogies into knowledge graphs (KGs) for a more machine-friendly format. We compare the knowledge graphs created by four relation extractors, three co-reference resolvers, and two embedding models via Pearson’s $r$ coefficient, root mean squared error (RMSE), and the Wilcoxon Signed-rank Test. We observe that the Wilcoxon Signed-rank Test provided the most streamlined result for algorithm and embedding selection, and that extractors were more influential than the resolver in creating KGs. From this, OpenIE was the best-performing extractor, and the SBERT embeddings yielded the KGs that best preserved analogical structure. Future work should focus on additional statistical tests and a greater range of information extraction algorithms and embeddings.