Dialogue discourse parsing aims to capture the relationships between utterances in a dialogue, facilitating a deeper understanding of dialogue structures and semantics, especially in long and complex dialogues. Existing research often develops separate dialogue discourse parsers for text-only and multimodal scenarios, largely due to the scarcity of parallel multimodal annotated datasets. This separation limits the ability to fully utilize diverse data with different modalities and poses challenges for real-world artificial intelligence applications.
To solve the problems, Doc. Gong Chen (Soochow University, China) and Yu Nan (Soochow University, China) and Professor Fu Guohong (Soochow University, China) published their new research on 15 September 2026 in Frontiers of Computer Science co-published by Higher Education Press and Springer Nature.
The team proposed a unified dialogue discourse parsing framework that bridges text-only and multimodal parsing within a single model. They propose a two-stage training method for the unified framework inspired by the success of training multimodal large language models. In the first stage, they develop a basic textual dialogue discourse parser and pre-train it on text-only datasets. In the second stage, they extend the model to accommodate multimodal scenarios by introducing a multimodal utterance representation module and a multimodal fusion module, and use the parameters learned from the first stage as initial parameters for the second stage, freezing them while training the multimodal parameters on multimodal datasets.
Extensive experiments on three datasets, covering both text-only and multimodal dialogues, show that the proposed approach achieves significant average improvements over several existing benchmarks. This demonstrates the generalizability and effectiveness of the unified framework for dialogue discourse parsing across different modalities.
Frontiers of Computer Science
Experimental study
Not applicable
Bridging modalities: a unified framework for textual and multimodal dialogue discourse parsing
15-Sep-2026