One-sentence Summary

Researchers from The Hong Kong University of Science and Technology (Guangzhou), Technical University of Munich, and collaborators propose HAM-RAG, a hierarchy-aware multimodal retrieval-augmented generation framework that uses document hierarchy as a grounding signal across retrieval and generation to preserve source position and local text-image relations, and introduce HAM-Bench, with HAM-RAG improving the main multimodal average by $17.3\%$17.3% over the strongest non-hierarchical baseline across multiple backbones for technical manuals and industrial SOPs.

Key Contributions

• The paper introduces HAM-RAG, a hierarchy-aware multimodal retrieval-augmented generation framework that uses document hierarchy to contextualize textual and visual evidence and preserve source position and local text-image relations in interleaved generation.
• The paper constructs HAM-Bench, covering Wukong, Wiki, arXiv, and Recipe across game walkthroughs, web pages, scientific papers, and step-wise recipe documents with local text-image alignment metadata.
• The paper designs an evaluation protocol and reports that HAM-RAG improves the main multimodal average by 17.3% over the strongest non-hierarchical baseline and improves Img-CBS by 24.2% on Wukong, showing stronger image selection, placement, and local text-image alignment.

Introduction

Multimodal retrieval-augmented generation (RAG) grounds language models in external text and visual sources, but many real-world sources are structured documents such as scientific papers, recipes, and game walkthroughs, where section hierarchy, procedural order, and local text-image relationships determine whether evidence is used correctly. Prior multimodal RAG approaches largely retrieve flat passages and images, so they often preserve only global semantic relevance and can select plausible images with weak local alignment or incorrect placement. The authors address this by formulating structure-faithful multimodal RAG and proposing HAM-RAG, a hierarchy-aware framework that encodes heading and section context for text evidence and document position, captions, nearby text, and context-aware visual descriptions for images; they also introduce HAM-Bench to evaluate retrieval sufficiency, image selection, image-context alignment, and answer quality over structured documents.

Dataset

Dataset: HAM-Bench

• Composition and sources

  • HAM-Bench is a hierarchy-preserving multimodal benchmark with four structured document subsets: Wukong, Wiki, arXiv, and Recipe.
  • Wukong is built from publicly available Black Myth: Wukong walkthroughs. It naturally includes section-level hierarchy, dense interleaved visual evidence, and requires multi-document reasoning.
  • Wiki is traced back to WikiWeb2M pages. It has shallower hierarchies and fewer images.
  • arXiv is built from original PDFs parsed into section-structured Markdown using MinerU. It has deep section hierarchies and dense visual content.
  • Recipe is built from source recipe documents reorganized into step-wise hierarchical procedures. It emphasizes procedural order and local text-image alignment.

• Processing and structure restoration

  • The authors preserve document hierarchy explicitly rather than flattening documents.
  • For reconstructed subsets, they restore structures already present in the original sources, including headings, section boundaries, figure-caption links, and procedural steps.
  • When source documents were originally flattened in prior resources, HAM-Bench reconstructs document boundaries, section structures, and local text-image alignments.
  • The benchmark supports hierarchy-aware retrieval and interleaved text-image generation.

• How the data is used

  • HAM-Bench is used to evaluate whether multimodal retrieval-augmented generation can preserve contextual organization and text-image logic in structured documents.
  • It is designed for structure-aware multimodal RAG evaluation under a shared retrieval-and-generation setting.
  • The provided material does not specify a training split or mixture ratios; it describes HAM-Bench as an evaluation benchmark rather than a training dataset.

Method

The authors propose HAM-RAG, a framework for retrieval-augmented generation over hierarchical multimodal documents. Unlike flat RAG pipelines that treat passages and images as isolated chunks, HAM-RAG leverages document hierarchy as a shared signal for evidence representation, retrieval, and prompt construction. The framework operates in two main stages: an offline hierarchy-aware indexing stage and an online multimodal retrieval and generation stage.

Hierarchy-Aware Document Representation Let $\mathcal{D} = \{D_1, ..., D_N\}$D={D1​,...,DN​} denote a corpus of structured multimodal documents. Each document is parsed into a hierarchy tree $T_D = (V_D, E_D)$TD​=(VD​,ED​), where internal nodes represent the document, sections, subsections, or procedural steps, and leaf nodes represent retrievable text and image units. Edges encode parent-child relations. The hierarchy is recovered from document markup, heading levels, numbering patterns, figure-caption links, layout cues, and source order.

For each retrievable leaf unit, the framework constructs a structured evidence object:

$$e_i = \{x_i, m_i, p_i, a_i, l_i, r_i\}$$ei​={xi​,mi​,pi​,ai​,li​,ri​}

where $x_i$xi​ is the original text or image content, $m_i$mi​ is its modality, $p_i$pi​ is the path from the document root to the unit, $a_i$ai​ contains ancestor headings and section context, $l_i$li​ contains nearby text-image context, and $r_i$ri​ records the unit's local relation to surrounding evidence. For instance, $r_i$ri​ may indicate that an image illustrates the current procedural step or supports the preceding paragraph. This representation makes the semantic scope and local role of each evidence unit explicit before retrieval.

Offline Evidence Construction and Indexing In the offline stage, the framework processes text and image units to build a hierarchy-aware vector store. For a text unit, HAM-RAG serializes the original content together with its document title, heading path, section context, and neighboring evidence into a hierarchy-aware textual representation. This preserves source wording while exposing the structural context needed to distinguish passages that are topically similar but belong to different sections or steps.

For an image unit, the framework retains its source position, heading path, caption or alt text, and nearby textual context. It additionally generates a context-aware visual description conditioned on both the image and its surrounding document evidence. This description supplements sparse or missing captions and helps connect visual content to the section or procedure in which it appears. Text and image evidence are then serialized into field-structured embedding prompts, embedded, and stored in a shared vector store together with provenance metadata such as document identifier, source position, modality, and image reference. Consequently, hierarchy affects retrieval through the indexed representation rather than being appended only after retrieval.

Online Retrieval and Interleaved Generation In the online stage, the framework receives a user query and retrieves the top-ranked textual and visual evidence under separate budgets $k_t$kt​ and $k_v$kv​. HAM-RAG retains each hit's relevance score, source identifier, document position, hierarchy-aware content, and local text-image relation. Keeping these fields allows evidence from different documents or sections to remain distinguishable during generation.

The selected evidence is assembled into a structured prompt:

$$P_{\text{ham}} = \text{Assemble}(q, \mathcal{E}^*)$$Pham​=Assemble(q,E∗)

where $\mathcal{E}^*$E∗ contains the ranked text and image evidence. Text blocks expose source-grounded headings and neighboring context, while image blocks include the image reference, caption or alt text, source position, and context-aware description. The generator therefore receives not only relevant content but also the local organization needed to decide which image supports which statement and where that image should be placed. In this way, hierarchy participates twice: it shapes the retrieval representation offline and guides evidence organization and interleaved text-image generation online.

Experiment

The experiments evaluate HAM-RAG on the HAM-Bench benchmark, which spans four structured document types with complementary challenges, and compare it against LLM-based, MLLM-based, and rule-based baselines under the same retrieval and generation budget. HAM-RAG consistently improves retrieval and multimodal generation across multiple backbone models, with the clearest gains in visual grounding and local text-image alignment, while ablations show that both image-side and text-side hierarchy contribute. Robustness results further indicate that hierarchy-aware structuring achieves very few failures and reasonable inference cost relative to other generation pipelines.

HAM-Bench is the only compared benchmark that fully supports multimodal evidence, interleaved text-image answer generation, explicit document structure preservation, reconstruction from flattened sources, and hierarchy-aware evaluation. Existing multimodal QA and RAG benchmarks either lack these features or support structure and reconstruction only partially, and none include hierarchy-aware evaluation. The cited experimental results also show that hierarchy-aware retrieval and generation improve visual grounding and local text-image alignment over non-hierarchical baselines. HAM-Bench is the only benchmark in the comparison with full support for structure preservation, reconstruction from flattened sources, and hierarchy-aware evaluation. Prior benchmarks have at most partial structure or reconstruction support and no hierarchy-aware evaluation; hierarchy-aware generation also improves visual grounding and local text-image alignment.

HAM-Bench spans four document types with complementary structural profiles. Academic documents have deep section hierarchies and dense visual content, game walkthroughs combine multi-document reasoning with procedural alignment, recipes provide large-scale procedural examples, and web pages are comparatively sparse in visual evidence. The academic subset is the most visually dense and structurally deep, with many embedded images per document and numerous lower-level headings. The game walkthrough subset requires multi-document and procedural reasoning and provides the highest image evidence per answer. The recipe subset contributes the largest number of documents and questions while maintaining moderate procedure-oriented image use. The web subset has the sparsest images per document and about one image per answer, making image selection less discriminative.

HAM-RAG improves multimodal generation over non-hierarchical baselines across all matched generator backbones, with the average MM Avg. rising from 54.68 to 64.12, a 17.3% relative improvement. Gains are strongest for local image-context alignment, especially on Wukong, while Wiki is less discriminative for image selection because single-reference-image questions often saturate Img-F1. Overall, hierarchy-aware evidence organization strengthens visual grounding and local text-image alignment. Across all nine matched backbones, HAM-RAG achieves a higher MM Avg. than its LLM-Based counterpart, with a 17.3% relative average improvement. The strongest local alignment gain appears on Wukong, where the best Img-CBS improves by 24.2% over the strongest non-hierarchical baseline. Recipe and arXiv gains indicate that structural context helps associate images with relevant procedural or sectional text rather than merely retrieving topically related figures. Wiki is less discriminative for image selection because single-reference-image questions frequently saturate Img-F1, so its results should be read with Img-CBS and quality scores.

HAM-RAG improves text retrieval over the shared baseline on every evaluated subset and improves image retrieval on three of four subsets. The only exception is arXiv, where flat retrieval achieves higher image recall by retaining more figures in an image-dense setting; this does not undermine the method's stronger local text-image alignment. Wiki image retrieval is less discriminative because single reference images often saturate image-level coverage, so its results are better interpreted with other visual grounding metrics. HAM-RAG outperforms the shared text retrieval baseline on all four subsets, while also achieving better image retrieval on Wukong, Wiki, and Recipe. The only image retrieval exception is arXiv, where the shared baseline's flat retrieval returns more figures, though recall does not measure whether those figures are placed with the correct local text.

HAM-Bench is the only evaluated benchmark with full support for multimodal evidence, interleaved text-image answers, document structure preservation, reconstruction from flattened sources, and hierarchy-aware evaluation, and it spans four complementary document types. Experiments compare hierarchy-aware HAM-RAG against non-hierarchical baselines across these subsets and show consistent gains in multimodal generation and visual grounding, particularly for local text-image alignment and procedural or sectional image association. Text retrieval improves on every subset, while image retrieval improves on three of four subsets; the only exception is image-dense arXiv, where flat retrieval returns more figures but does not ensure those figures are aligned with the correct local text.