One-sentence Summary

Researchers from the University of North Carolina at Chapel Hill and Apple propose CoGR, a retrieval framework where LLMs generate keywords for both queries and items, matched via an inverted index, and co-evolve through reinforcement learning with GRPO, optimizing retrieval $F _ { 1 }$F1​ via direct and counterfactual marginal rewards, achieving state-of-the-art $F _ { 1 }$F1​ improvements of 10.9% on an APP Marketplace dataset and 36.1% on WANDS.

Key Contributions

• CoGR is a generative retrieval framework where LLMs directly produce keyword sets for both queries and items, enabling end-to-end matching through an inverted index without a downstream retriever.
• The framework uses a two-stage training pipeline that first aligns keyword spaces via supervised fine-tuning, then alternately optimizes query and item generators with GRPO reinforcement learning against a frozen opposite-side index under a shared retrieval F1 objective.
• Experiments on an internal APP Marketplace dataset and the public WANDS benchmark show that CoGR outperforms ten sparse, dense, and generative baselines, improving F1 by 10.9% and 36.1% over the strongest baseline, with analysis confirming stable co-evolution and increasingly aligned query-item keyword spaces.

Introduction

Retrieval is the critical first stage in search and recommendation systems, where errors are irreversible: missing an item cannot be recovered later, while irrelevant candidates burden downstream ranking. Classical lexical methods like BM25 rely on exact term matches but struggle with deeper semantic relationships, and dense retrieval maps queries and items into continuous vector spaces, while generative retrieval predicts identifiers autoregressively but faces scalability and generalization challenges. Recent work uses large language models (LLMs) to improve retrieval through query expansion or keyword generation, yet these approaches typically train only the query side and still depend on a separate retriever for matching. The authors propose CoGR, a co-evolving generative retrieval framework that trains separate LLMs to generate compact keyword sets for both queries and items, enabling direct matching via an inverted index. They address the core challenge of aligning the two keyword spaces through a two-stage pipeline: supervised fine-tuning initializes aligned representations, and alternating reinforcement learning with GRPO optimizes each side against a frozen opposite-side index using a shared retrieval F1 reward, allowing the keyword spaces to co-adapt progressively.

Dataset

The authors use two industrial search datasets, both containing many relevant items per query to better reflect practical retrieval settings.

• Dataset sources and composition

  • Internal APP marketplace dataset: de-identified, randomly sampled user queries; each item is an application represented by its title and description.
  • WANDS (Wayfair): a public product-search dataset where each item is a product, also represented by title and description.

• Key details for each subset

  • Both datasets provide categorical relevance annotations for (query, item) pairs.
  • Annotations are binarized into relevant and irrelevant classes (details in Appendix A).
  • The full item universe is retained for both training and validation.
  • Dataset statistics (including the number of relevant items per query) are summarized in Table 1 of the paper.
  • No explicit size or filtering rules beyond binarization are described in the text; the internal dataset is randomly sampled, and WANDS is used as-is.

• Data processing and usage

  • The data is split only along the query dimension, so training and validation sets contain disjoint queries but share the same item catalog.
  • This split ensures that validation performance reflects generalization to unseen queries.
  • No cropping or additional metadata construction is mentioned; items are used with their title and description text.
  • The paper does not specify mixture ratios or other training-split details beyond the query-based split.

Method

The authors propose CoGR, a retrieval framework that generates keywords for both queries and items to perform keyword-based matching. Given a query $q \in \mathcal{Q}$q∈Q and an item $i \in \mathcal{I}$i∈I, two separate keyword generators, $G^q$Gq and $G^i$Gi, produce keyword sets $S_q = G^q(q)$Sq​=Gq(q) and $S_i = G^i(i)$Si​=Gi(i). The retrieved item set is defined as $I_{\mathrm{ret}}(q) = \{ i: (S_q \cup \{q\}) \cap (S_i \cup \{i\}) \neq \emptyset \}$Iret​(q)={i:(Sq​∪{q})∩(Si​∪{i})=∅}. To enable retrieval ranking, the keyword sets are treated as bags of words, and items are ranked using BM25 scores.

The training pipeline consists of two stages: a supervised fine-tuning (SFT) stage to initialize the generators, followed by a reinforcement learning (RL) stage to optimize retrieval quality. The alternating training paradigm of the RL stage is illustrated in the figure below:

Phase 1: Initialization with Supervised Fine-Tuning The SFT stage establishes an aligned keyword space and provides a meaningful initialization for the RL phase. For each item $i$i, the original LLM generates an initial item-side keyword set $S_i$Si​. To construct query-side targets, the authors collect relevant items for each query $q$q, pool their initial item-side keywords, and select the top-$N$N most frequent keywords as the target set $S_q$Sq​. This ensures keyword overlap between relevant query-item pairs. The query-side generator $G^q$Gq and item-side generator $G^i$Gi are then trained on $\{(q, S_q)\}${(q,Sq​)} and $\{(i, S_i)\}${(i,Si​)} respectively, yielding initialized policies $G_{\mathrm{SFT}}^q$GSFTq​ and $G_{\mathrm{SFT}}^i$GSFTi​.

Phase 2: Co-Evolving Reinforcement Learning Following initialization, the RL stage optimizes the generators directly for retrieval quality using an alternating training paradigm. The query-side and item-side generators are updated in turn, while the inverted index from the opposite side remains frozen. This allows each generator to optimize against a fixed retrieval environment.

Query-Side RL Given the frozen item indexes, the query-side generator produces a keyword set $S_q$Sq​ for each query $q$q. The retrieved set $I_{\mathrm{ret}}(q)$Iret​(q) is evaluated using the $F_1$F1​ score to balance precision and recall:

$$P(q) = \frac{|\mathrm{rel}(q) \cap I_{\mathrm{ret}}(q)|}{|I_{\mathrm{ret}}(q)|}, \quad R(q) = \frac{|\mathrm{rel}(q) \cap I_{\mathrm{ret}}(q)|}{|\mathrm{rel}(q)|}, \quad F_1(I_{\mathrm{ret}}(q), \mathrm{rel}(q)) = \frac{2P(q)R(q)}{P(q) + R(q)}$$P(q)=∣Iret​(q)∣∣rel(q)∩Iret​(q)∣​,R(q)=∣rel(q)∣∣rel(q)∩Iret​(q)∣​,F1​(Iret​(q),rel(q))=P(q)+R(q)2P(q)R(q)​

To constrain the output length, a maximum keyword budget $K_{\mathrm{max}}$Kmax​ is imposed. The reward function is defined as:

$$\mathcal{R}_q(S_q) = \begin{cases} F_1(I_{\mathrm{ret}}(q), \mathrm{rel}(q)), & |S_q| \leq K_{\mathrm{max}} \\ 0, & |S_q| > K_{\mathrm{max}} \end{cases}$$Rq​(Sq​)={F1​(Iret​(q),rel(q)),0,​∣Sq​∣≤Kmax​∣Sq​∣>Kmax​​

Following the GRPO paradigm, multiple keyword sets are sampled for each query, and the rewards are normalized within each rollout group to compute relative advantage signals for optimization.

Item-Side RL Item-side RL refines item keywords by evaluating the marginal contribution of a candidate keyword set $S_i$Si​ to the overall retrieval quality. At the start of each update round, the query-side index is frozen. A counterfactual index is constructed by replacing only the reference keywords of item $i$i with $S_i$Si​. The item-side reward is defined as the difference in aggregate retrieval quality between the counterfactual and reference indexes:

$$\mathcal{R}_i(S_i) = \begin{cases} \sum_{q \in \mathcal{Q}} F_1(I_{\mathrm{ret}}^{\mathrm{cand}}(q; S_i), \mathrm{rel}(q)) - \sum_{q \in \mathcal{Q}} F_1(I_{\mathrm{ret}}^{\mathrm{ref}}(q), \mathrm{rel}(q)), & \text{if } |S_i| \leq K_{\mathrm{max}} \\ -1, & \text{otherwise} \end{cases}$$Ri​(Si​)={∑q∈Q​F1​(Iretcand​(q;Si​),rel(q))−∑q∈Q​F1​(Iretref​(q),rel(q)),−1,​if ∣Si​∣≤Kmax​otherwise​

This difference-based formulation isolates the effect of $S_i$Si​ and enables efficient computation by only evaluating queries whose $F_1$F1​ scores change.

Co-Evolving Process The RL phase iteratively alternates between query-side and item-side optimization. Starting from the post-SFT models, each side is optimized against an index built by the latest model of the other side. This co-evolution process allows the two generators to progressively adapt to each other's keyword space, jointly evolving to improve overall retrieval quality. The framework remains flexible, allowing the $F_1$F1​ reward to be replaced by a weighted F-measure if precision and recall have different business priorities.

Experiment

The method is evaluated on two industrial search datasets with binarized relevance, comparing against sparse, dense, and generative retrieval baselines. Experiments show that the co-evolving reinforcement learning process jointly aligns query and item keyword spaces, yielding the strongest and most consistent retrieval performance. Ablations confirm that each component (marginal item-side reward, separate generators, SFT initialization) contributes to the gains, while keyword evolution analysis reveals a shift toward more specific, multi-word phrases and balanced vocabularies. The framework also benefits from richer textual context, such as item descriptions and external search results.

The internal app marketplace dataset is substantially larger in query volume, with 13,500 training queries compared to 430 in WANDS, while both datasets cover a similar item universe of roughly 40,000 items. The internal dataset also exhibits a much higher density of relevant items per query, averaging around 1,000, versus approximately 200 for WANDS. The internal dataset contains roughly five times more relevant items per query than WANDS (≈1,000 vs ≈200). Both datasets have comparable item universes of about 40,000 items, but the internal dataset provides over 30 times more training queries.

CoGR achieves the highest overall F1 on both validation datasets, surpassing all baselines. Jointly evolving query and item keyword spaces is crucial, as CoGR substantially outperforms variants that optimize only the query side. Among baselines, dense retrieval with ANCE-Qwen4B is the most consistent, while sparse and generative methods show clear dataset-specific weaknesses. CoGR attains the best overall F1 on Internal (0.396) and WANDS (0.682). Co-evolving query and item keywords yields large gains over query-only optimization, seen by CoGR's lead over CoGR* and DeepRetrieval. Dense retrieval with ANCE-Qwen4B is the strongest baseline, while sparse retrieval struggles on the Internal dataset and generative retrieval degrades on WANDS.

The full CoGR model achieves the highest retrieval F1 among all configurations. Removing the marginal item-side reward, sharing the generator, or skipping SFT initialization each degrades performance, but all variants still yield reasonable results, demonstrating the robustness of the co-evolving framework. The full CoGR model attains the best F1 score, outperforming all ablated variants. Sharing a single generator for both query and item sides reduces retrieval performance compared to using separate generators. Skipping the SFT initialization phase and starting RL directly from the base model lowers F1, confirming the benefit of warm-start training. Replacing the marginal item-side reward with a symmetric item-centric objective decreases precision and F1. All ablated variants maintain stable and reasonable retrieval performance, indicating the framework is resilient to these design changes.

Enriching the textual context for keyword generators consistently improves retrieval performance. Removing item descriptions reduces F1, while adding search results to the query-side prompt substantially raises both precision and recall, mainly by helping resolve ambiguous or non-standard queries. Removing item descriptions drops F1 from 0.3963 to 0.3759, showing that descriptions provide important cues when titles alone are underspecified. Adding search results lifts F1 to 0.4379, with the largest gains coming from better handling of ambiguous, misspelled, entity-centric, or non-English queries.

Experiments on an internal app marketplace dataset and the public WANDS dataset show that CoGR, which jointly evolves query and item keyword spaces via reinforcement learning, achieves the highest retrieval F1, outperforming dense, sparse, and generative baselines. Co-evolving both sides is essential, as query-only optimization lags significantly, while adding item descriptions and search result context further boosts performance. Ablations confirm the robustness of the framework, with all variants maintaining reasonable results even when individual components are removed.