One-sentence Summary

Researchers at Wuhan University, Monash University, and RMIT University propose DiscoSG-Refiner, a lightweight open-source parser for discourse-level scene graph parsing that drafts a seed graph and iteratively refines it via a learned graph-editing model, achieving 30% higher SPICE than baselines and $86\times$86× faster inference than GPT-4o while improving downstream VLM caption evaluation and hallucination detection.

Key Contributions

• The paper introduces DiscoSG, a discourse-level text scene graph parsing task, and releases DiscoSG-DS, a dataset of 400 expert-annotated and 8,430 synthesised multi-sentence caption-graph pairs. Each caption averages 9 sentences, and each graph contains at least 3 more triples than those in existing datasets.

• The paper presents DiscoSG-Refiner, a 0.5B open-source iterative parser that drafts a seed graph and refines it with a learned graph-editing model trained on DiscoSG-ED, a synthetic corpus of edit operations derived from DiscoSG-DS. The model achieves 30% higher SPICE than the best sentence-merging baseline on simple and dense graphs while using substantially lower inference cost than fine-tuned GPT-4o.

• The paper establishes D-FOIL, a benchmark of 200 pairs of hallucinated and human-corrected multi-sentence captions for discourse-level hallucination detection. DiscoSG-Refiner improves downstream VLM tasks, including discourse-level caption evaluation and hallucination detection, over sentence-merging and end-to-end Flan-T5 parsers.

Introduction

Text scene graph parsing converts visual descriptions into graphs of entities and their relations, supporting tasks such as image captioning evaluation, hallucination detection, and image retrieval. The problem has become more pressing because vision language models now produce multi-sentence captions with cross-sentence coreference, long-range dependencies, implicit relations, and requirements for global graph coherence. Existing sentence-level parsers and heuristic merging methods often ignore these discourse phenomena, while fine-tuning large models is slow, expensive, and licensing constrained, and smaller models degrade on dense graphs and downstream vision language tasks. The authors define the DiscoSG task, release the DiscoSG-DS dataset of 400 expert-annotated and 8,430 synthesized caption-graph pairs, introduce the D-FOIL benchmark for discourse-level hallucination detection, and propose DiscoSG-Refiner, a lightweight iterative refinement framework built from Flan-T5 editors trained on synthetic edit operations to generate coherent scene graphs more efficiently.

Dataset

• The authors introduce DiscoSG-DS, a discourse-level scene graph dataset in which each example pairs a multi-sentence image description with a scene graph. The descriptions contain contextually dependent sentences, and the graphs are represented as structured triples for object relationships, such as man, wear, hat, and attributes, such as hat, has_attribute, red.

• The source pool comes from SharedGPT4V, a collection of about 1.2 million GPT-4 generated image captions. The authors select 40,000 diverse image-description pairs using TF-IDF text embeddings and diversity sampling, then route selected examples to manual annotation or synthetic annotation.

• The manual gold subset contains 400 annotated examples randomly selected from the 40,000 candidate pairs. Annotation uses a human-in-the-loop active learning pipeline. Annotators first produce FACTUAL-MR intermediate annotations and then convert them into final scene graphs, resolving cross-sentence coreference, quantifier inconsistencies, long-range relations, implicit relations, semantic contradictions, and image-verified hallucinations.

• In the manual pipeline, 102 initial descriptions are used to build draft graphs from sentence-level parses. Forty examples are reserved as a fixed validation set, while 62 become the seed training set. Two active learning rounds add refined human corrections from batches of 94 and 204 examples, expanding the training pool to 360 examples. The validation set is then merged with the training set, and the 400 human-annotated graphs are used to create 300 training and 100 test instances, described as two 300/100 train-test splits.

• The synthesized subset contains 8,430 examples. The authors first filter descriptions by comparing them with their images using GPT-4o and remove hallucination-prone descriptions. They then annotate the remaining descriptions with a GPT-4o teacher model fine-tuned on the 300 human-annotated training examples.

• The final dataset has 300 human-annotated training examples, 100 human-annotated test examples, and 8,430 synthesized examples. Descriptions average 9.3 sentences. Each graph contains roughly 15 more triples than Visual Genome or FACTUAL examples and 3 more triples than TSGBench examples, reflecting longer discourse spans and more complex relational structure.

• For downstream use, the authors derive an edit-annotated dataset, DiscoSG-ED, from DiscoSG-DS. For each manual or synthetic instance, they compare a generated graph with the gold graph to create ground-truth deletion and insertion annotations. They also create extra edit targets by randomly deleting or inserting triples in the gold graph. Each training item becomes a tuple of description, initial graph, deletion flags, and insertion sequence. With N corruptions per gold graph, DiscoSG-ED scales to N times the size of DiscoSG-DS.

Method

The method combines a scalable annotation workflow with a three module refinement architecture for discourse level scene graph parsing. Its central premise is that a flawed initial graph can be corrected through targeted edit actions rather than being regenerated entirely. This design reduces decoding burden for small pretrained language models while preserving the ability to capture inter-sentence coreference, implicit relations, quantifier corrections, and other dependencies that sentence level parsers often miss.

The authors leverage a human in the loop annotation pipeline to construct reliable supervision for model training and teacher model construction. Descriptions are first parsed into a structured intermediate representation that reduces ambiguity and standardizes the annotation format. During manual annotation, reviewers focus on resolving cross-sentence coreference, correcting quantifier inconsistencies, recovering missing long-range relations, identifying implicit relations, and detecting semantic contradictions. The protocol also requires annotators to verify descriptions against the corresponding images and correct factual inconsistencies or hallucinations before the graphs are finalized.

This annotation process is organized as an iterative active learning loop. The process begins with draft scene graphs generated from sentence level parses and refined through staged human review. A portion of these curated examples is reserved as a fixed validation set, while the remainder forms a small seed training set. A base language model is then fine tuned on the seed set to produce an initial teacher model. In each active learning round, the model drafts scene graphs for newly sampled descriptions, a less senior annotator edits the drafts, and a senior annotator validates them. The newly accepted graphs are added to the training set, and the model is fine tuned again. This cycle repeatedly directs annotation effort toward instances that improve model behavior while keeping human cost controlled. In the reported workflow, two rounds were sufficient because validation gains plateaued and the annotation budget was fixed.

For larger scale supervision, the authors use the manually supervised data as a bridge to synthetic annotation. A model is first used to filter descriptions that exhibit hallucination errors when compared with their images, retaining only descriptions with reliable visual grounding. A teacher model, fine tuned on the human annotated training examples, then generates scene graph annotations for the remaining descriptions. This creates a larger corpus of model produced labels whose quality is grounded in high-quality human examples. These labels support downstream training without requiring exhaustive manual review of all scaled examples.

The parsing model itself is formalized as learning a most probable scene graph for a multi-sentence input. Given a discourse description $\boldsymbol{x}$x, the target output is a graph $\boldsymbol{y}$y, and inference can be written as

$$\boldsymbol{y}^* = \arg\max_{\boldsymbol{y} \in \mathcal{Y}} P(\boldsymbol{y} \mid \boldsymbol{x}; \boldsymbol{\theta}),$$y∗=argy∈Ymax​P(y∣x;θ),

where $\mathcal{Y}$Y is the space of possible scene graphs and $\boldsymbol{\theta}$θ denotes model parameters. The refinement model does not produce this graph in one autoregressive pass. Instead, it starts from an initial graph and applies a sequence of edit operations. The probability of the final graph after $T$T refinement steps is decomposed as

$$P(\boldsymbol{y}^T \mid \boldsymbol{x}; \boldsymbol{\theta}) = P_{\mathrm{Gen}}(\boldsymbol{y}^0 \mid \boldsymbol{x}) \prod_{t=0}^{T-1} \left( P_{\mathrm{Prog}}(\boldsymbol{a}^t \mid \boldsymbol{y}^t, \boldsymbol{x}; \boldsymbol{\theta}) \cdot P_{\mathrm{Intp}}(\boldsymbol{y}^{t+1} \mid \boldsymbol{y}^t, \boldsymbol{a}^t) \right).$$P(yT∣x;θ)=PGen​(y0∣x)t=0∏T−1​(PProg​(at∣yt,x;θ)⋅PIntp​(yt+1∣yt,at)).

This equation reflects three modules: a Generator, a Programmer, and an Interpreter. The Generator proposes an initial graph, the Programmer proposes edit actions, and the Interpreter deterministically applies those actions to transform the graph.

The Generator provides broad coverage of explicit mentions while intentionally accepting errors that the refinement modules can later correct. It operates by applying a sentence level parser independently to each sentence in the input discourse. Each sentence is converted into a sentence level graph, and these partial graphs are merged into an initial discourse graph. During merging, nodes with identical names are treated as coreferent, which gives the initial graph a reasonable starting point for later relation resolution. However, because this stage processes sentences mostly independently, it often produces redundant triples, missing cross-sentence links, and omitted implicit relations. These deficiencies are not treated as fatal failures; they are the input state for the refinement process.

The Programmer is the core learnable module. At each refinement step $t$t, it receives the current graph $\boldsymbol{y}^t$yt and the original discourse $\boldsymbol{x}$x, and it predicts an action set

$$\boldsymbol{a}^t = (\mathbf{D}^t, \mathbf{I}^t),$$at=(Dt,It),

where $\mathbf{D}^t$Dt denotes deletion actions and $\mathbf{I}^t$It denotes insertion actions. This separation is important. Generating deletion and insertion operations together can make the decoder output too long for small language models, while insert-only editing cannot remove redundant or incorrect triples already present in the initial graph. Empirical observation of the initial graphs motivates the design: a substantial fraction of proposed triples must be deleted, so a deletion-aware architecture is necessary.

The Programmer uses an encoder decoder pretrained language model with the deletion and insertion tasks disentangled. The encoder predicts deletion flags. The current graph is represented in a flattened form, and each triple is embedded by pooling contextual token representations between its opening and closing boundaries. A binary classification layer then labels each triple as either a triple to keep or a triple to delete. This turns large scale pruning of the initial graph into an efficient classification problem rather than a generative problem.

The decoder handles insertions separately. Conditioned on the encoder representation of $\boldsymbol{x}$x and $\boldsymbol{y}^t$yt, it generates token sequences that correspond to new graph triples. Each generated token sequence implicitly represents an insertion action. Because the decoder only generates missing content rather than reconstructing the entire graph, the required generation length is substantially reduced. This also keeps the architecture compatible with smaller open source language models while still enabling iterative graph improvement.

The Interpreter makes the editing process deterministic and structurally controlled. It receives the current graph and the predicted edit actions, then produces the next graph state. First, it applies deletions by removing triples flagged by the encoder. Second, it parses the decoder output into candidate triples, validates them for structural correctness, and discards malformed or incomplete triples. Valid triples are then added to the graph. The interpreter follows a delete first then insert ordering, which can be written as

$$\boldsymbol{y}^{t+1} = (\boldsymbol{y}^t \setminus \mathcal{D}^t_{\mathrm{triples}}) \cup \mathcal{T}^t_{\mathrm{triples}}.$$yt+1=(yt∖Dtriplest​)∪Ttriplest​.

Because the Interpreter is deterministic, $P_{\mathrm{Intp}}(\boldsymbol{y}^{t+1} \mid \boldsymbol{y}^t, \boldsymbol{a}^t)$PIntp​(yt+1∣yt,at) is one for the unique resulting state and zero otherwise.

Training the Programmer requires edit supervision rather than only final graph supervision. The authors construct edit annotated data by comparing an initial generated graph with the corresponding gold graph. For each description and gold graph pair, the Generator produces an initial graph. Gold graph perturbations are also created by randomly deleting or inserting triples, producing alternative initial states that simulate realistic errors. Ground truth deletion flags are obtained by identifying triples present in the initial graph but absent from the gold graph. Ground truth insertion sequences are obtained by identifying triples present in the gold graph but absent from the initial graph and converting those triples into token sequences. This yields training examples of the form

$$(\boldsymbol{x}, \boldsymbol{y}^0, \mathbf{D}_{gt}, \mathbf{I}_{gt}),$$(x,y0,Dgt​,Igt​),

where $\boldsymbol{x}$x is the discourse input, $\boldsymbol{y}^0$y0 is the initial graph, $\mathbf{D}_{gt}$Dgt​ is the deletion target, and $\mathbf{I}_{gt}$Igt​ is the insertion target. Synthetic corruption increases the number of edit examples because multiple corrupted initial graphs can be derived from the same gold annotation.

The encoder and decoder are jointly trained through multitask learning. The deletion module uses a binary cross entropy objective over keep or delete flags:

$$\mathcal{L}_{\mathrm{delete}}.$$Ldelete​.

The insertion module uses a sequence to sequence cross entropy objective for generating the target insertion sequence:

$$\mathcal{L}_{\mathrm{insert}}.$$Linsert​.

The total training loss is the weighted combination

$$\mathcal{L} = \lambda_{\mathrm{del}}\mathcal{L}_{\mathrm{delete}} + \lambda_{\mathrm{ins}}\mathcal{L}_{\mathrm{insert}}.$$L=λdel​Ldelete​+λins​Linsert​.

This training setup teaches the Programmer two complementary abilities. The encoder learns which existing graph elements are likely to be incorrect or redundant, while the decoder learns what discourse level relations are missing and how to express them as graph triples. At inference time, the model repeatedly proposes deletions and insertions, the Interpreter executes them, and the graph evolves toward a more coherent discourse level representation.

Experiment

The experiments evaluate DiscoSG-Refiner on discourse-level scene graph parsing, downstream VLM evaluation tasks, and component ablations. They show that refinement-based parsing improves accuracy and generalization to complex graphs while offering a stronger efficiency-quality tradeoff than many baseline methods. The qualitative results further indicate that DiscoSG-Refiner produces higher-quality scene graphs that make graph-based metrics more effective for caption assessment and hallucination detection. Ablations confirm that synthetic data, augmentation, multiple refinement steps, and a disentangled edit design are key to the method’s performance.

The illustration shows how an initial scene graph derived from a multi-sentence caption is revised through deletion and insertion steps. The process removes unsupported or redundant relations and adds missing entity attributes, yielding a more coherent final graph. This supports the idea that discourse-level scene graph parsing benefits from refinement rather than simple merging of sentence-level outputs. Initial scene graphs capture entities, predicates, and relations from the caption before refinement. Deletion edits adjust questionable or redundant graph relations during the iterative process. Insertion edits add missing graph information, such as an attribute linking people to a group. The final refined graph is more coherent than the initial graph after edit-based correction.

The dataset comparison shows that the new discourse-level scene graph dataset is far more extensive than single-sentence caption datasets. Its human and synthetic subsets both contain much longer text and more triples, objects, and relations per instance. The synthetic portion contributes a large share of the overall triple count while having average graph statistics close to the human portion. Discourse-level instances are substantially denser and more complex than prior caption-only datasets. Human and synthetic subsets show similar average description length and graph complexity. The synthetic subset contributes a large portion of the total triples in the dataset.

Fine-tuned DiscoSG models improve discourse-level scene graph parsing over sentence merging and direct baselines, but the gains depend strongly on model size, architecture, and training strategy. Larger fine-tuned models generally perform better, while small end-to-end models can drop sharply when graph length or complexity changes. The iterative refinement variants give a strong accuracy, generalization, and cost balance, approaching or outperforming much larger baselines with lower computational overhead. Using DiscoSG training improves discourse parsing, especially when paired with fine-tuning. Scale helps encoder-decoder and large decoder models, but small fine-tuned models are sensitive to graph complexity. The refinement approach preserves strong performance across splits while using far fewer parameters than the strongest large baseline. Local open models provide a practical alternative to closed large models in speed and deployment cost.

The annotation results show that most parsers still make frequent discourse-level errors, with long-range dependencies, implicit inference, and graph coherence remaining especially difficult. DiscoSG-Refiner-large reduces the overall error rate compared with the sentence-merging baseline and a small end-to-end parser, while approaching the error levels of GPT-4o-based systems. Strong text-only and teacher variants show that general language ability and supervised refinement both help mitigate cross-sentence and coherence errors. Small end-to-end parsing suffers the highest error rates, especially on long-range and coherence-related categories. Iterative refinement lowers average discourse errors relative to sentence merging and direct small-model parsing. GPT-4o-based variants achieve the lowest average errors, with text-only GPT-4o particularly strong on graph coherence.

Replacing the FACTUAL-T5-base Sent+Merge scene graph source with DiscoSG-Refiner improves graph-based evaluation metrics across captioning and hallucination detection tasks, with the larger refinement model giving the strongest gains. The smaller DiscoSG-Refiner model also mostly exceeds the baseline, while DiscoSG-T5 parsers reduce performance. Graph-based metrics using DiscoSG-Refiner-large rank among the best automated metrics, with BSSPICE and CAPTURE standing out. Scene graphs from DiscoSG-Refiner increase correlation metrics on DetailCaps and CapArena. CAPTURE shows notable gains on CapArena and D-FOIL when using DiscoSG-Refiner scene graphs. BSSPICE achieves the best average rank among automated metrics and strong D-FOIL accuracy.

Experiments evaluate a discourse-level scene graph dataset and DiscoSG parsing methods against sentence merging and direct baselines. The dataset is substantially denser and more complex than prior caption-only resources, and iterative refinement improves graph coherence by removing unsupported or redundant relations and adding missing entity information. DiscoSG-Refiner reduces discourse-level errors, especially long-range dependencies and coherence issues, while offering a cost-effective alternative to large closed models and improving downstream graph-based captioning and hallucination evaluation metrics.