From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation
Abstract: Narratives in news discourse play a critical role in shaping public understanding of economic events, such as inflation. Annotating and evaluating these narratives in a structured manner remains a key challenge for Natural Language Processing (NLP). In this work, we introduce a narrative graph annotation framework that integrates principles from qualitative content analysis (QCA) to prioritize annotation quality by reducing annotation errors. We present a dataset of inflation narratives annotated as directed acyclic graphs (DAGs), where nodes represent events and edges encode causal relations. To evaluate annotation quality, we employed a $6 imes3$ factorial experimental design to examine the effects of narrative representation (six levels) and distance metric type (three levels) on inter-annotator agreement (Krippendorrf’s $lpha$), capturing the presence of human label variation (HLV) in narrative interpretations. Our analysis shows that (1) lenient metrics (overlap-based distance) overestimate reliability, and (2) locally-constrained representations (e.g., one-hop neighbors) reduce annotation variability. Our annotation and implementation of graph-based Krippendorrf’s $lpha$ are open-sourced. The annotation framework and evaluation results provide practical guidance for NLP research on graph-based narrative annotation under HLV.
Show BibTeX
@article{DBLP:journals/corr/abs-2603-01930,
author = {Junbo Huang and
Max Weinig and
Ulrich Fritsche and
Ricardo Usbeck},
title = {From Variance to Invariance: Qualitative Content Analysis for Narrative
Graph Annotation},
journal = {CoRR},
volume = {abs/2603.01930},
year = {2026},
url = {https://doi.org/10.48550/arXiv.2603.01930},
doi = {10.48550/ARXIV.2603.01930},
eprinttype = {arXiv},
eprint = {2603.01930},
timestamp = {Tue, 21 Apr 2026 16:23:24 +0200},
biburl = {https://dblp.org/rec/journals/corr/abs-2603-01930.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}