One-sentence Summary

Cornell Tech and Cornell University propose COGEN, which tightly couples language comprehension and generation to enable continual learning from user interaction feedback in two-player reference games, yielding up to $26\%$26% absolute performance gains over non-coupled systems and significantly more human-like language output.

Key Contributions

• The proposed system couples language comprehension and generation for continual learning from user interactions in two-player reference games, where an agent alternates between listening, speaking, and learning from interaction feedback. It performs coupling through approximate Rational Speech Acts joint inference and shared training data, using interaction feedback as rewards for contextual bandit learning over deployment rounds.

• Human-interaction experiments spanning thousands of deployments evaluate comprehension through target selection accuracy and generation through human accuracy in selecting targets from generated descriptions. The coupled system improves over time, with up to 26% absolute gains and up to 17% higher accuracies than a non-coupled system.

• Linguistic analyses show that comprehension-generation coupling changes the agent's generated language across learning rounds. The resulting language is significantly more similar to human language, indicating qualitative effects beyond accuracy gains.

Introduction

Language comprehension and generation are interdependent in humans, and computational systems could benefit if a model improves one capability by exercising the other. The authors leverage a continual learning setup in which a single vision-language model alternates between listener and speaker roles while interacting with human partners in a reference game over abstract visual stimuli. Earlier work often studied pragmatic reasoning, instruction learning, or RLHF style training in more static or separate settings, making it hard to observe how comprehension and generation reshape one another over time, and such tasks remain difficult for current models. The authors’ main contribution is to couple comprehension and generation at inference time through joint reasoning about the opposing role and at training time by converting human interaction feedback into examples and rewards for both roles, creating a virtuous cycle that improves performance, data efficiency, and similarity to human language.

Dataset

• The authors build the visual game pool from the KILOGRAM dataset, which contains 1,016 abstract tangram shapes.

• Each game context is constructed from 10 images sampled from KILOGRAM. To make contexts more challenging and visually coherent, they use a CLIP model finetuned on KILOGRAM annotations to encourage visual similarity among images within each context.

• Interaction data is collected through human model games on MTurk over four deployment rounds. For each system and round, the authors collect an equal number of listener and speaker role interactions.

• The interaction scale starts at 2,000 interactions per role in round 1 and increases by 500 per round, giving 2,000, 2,500, 3,000, and 3,500 role interactions across the four rounds.

• Before model interaction data is used, the authors initialize training with 104 successful human human games. They also reuse these examples in later rounds and assign them a reward of +1.

• They hold out 280 successful human human games as a validation set for model selection throughout the experiments.

• The online feedback process creates two data subsets per round: D_l for comprehension and D_s for generation.

• A comprehension datapoint contains the image context, a human utterance, the model predicted target image index, and a binary reward. The reward is +1 when the predicted target is correct and -1 otherwise.

• A generation datapoint contains the image context, a model generated utterance, the target image index, and a binary reward. The reward reflects whether the human listener correctly selected the target after hearing the model utterance.

• These datasets are reward-labeled model interaction logs, not static supervision samples and not external pairwise preference annotations.

• The successful human human data is used for initialization, validation, and as reward +1 examples in later continual training. No fixed mixture ratio is specified for combining online interaction data with the initial human human examples.

• No cropping strategy or additional image manipulation is described. The main dataset processing details are KILOGRAM based context construction, CLIP-guided similarity selection, and binary success-based reward labeling.

Method

The authors leverage a single autoregressive multimodal language model, IDEFICS2 8B, with parameters $\theta$θ, to act in two complementary roles. The interaction scenario is a two player reference game in which both participants observe a context of abstract tangram images $\mathcal{I} = \{I_1, \ldots, I_N\}$I={I1​,…,IN​}, typically in different orders. One player is the speaker, assigned a target index $t$t, and generates an utterance $u$u intended to identify the target image $I_t$It​ to the listener. The listener then selects one image from the context. The interaction succeeds when the selected image matches the intended target. As shown in the figure below:

The model instantiates the two roles with two parameterized distributions. In the speaker role, it models

the probability of generating utterance $u$u given the visual context $\mathcal{I}$I and target $t$t. In the listener role, it models

the probability of selecting target $t$t given the visual context and utterance $u$u. Both the utterance generation and the target selection are handled through an autoregressive decoding process, and the listener prediction can be written as

Learning proceeds in rounds. In round $\rho$ρ, the deployed model uses parameters $\theta_\rho$θρ​ to interact with human partners. When the model plays the listener, it receives a human generated utterance $u$u, predicts $\hat{t}$t^, and receives binary feedback indicating whether the selection was correct. This creates a comprehension datapoint

where $r = 1$r=1 for success and $r = -1$r=−1 for failure. When the model plays the speaker, it samples

and the human listener’s outcome provides the reward, yielding a generation datapoint

Each round produces datasets $\mathcal{D}_{l,\rho}$Dl,ρ​ for comprehension and $\mathcal{D}_{s,\rho}$Ds,ρ​ for generation. Across rounds, the method accumulates all feedback collected so far:

The model for the next round, $\theta_{\rho+1}$θρ+1​, is estimated from the original model weights using all accumulated interaction data.

The training objective is formulated as a multi task contextual bandit problem combining comprehension and generation. The authors use a REINFORCE style policy gradient update. For a comprehension example $(\mathcal{I}, u, \hat{t}, r)$(I,u,t^,r) collected at an earlier round $m$m, the gradient is

with a case based inverse propensity coefficient

Here $\theta_m$θm​ denotes the parameters used when the interaction was originally generated. This coefficient reduces the influence of failed examples as their predicted probability decreases, preventing the logarithmic loss from becoming unbounded and destabilizing training. The generation gradient has the same form, but uses $P_s(\hat{u} \mid \mathcal{I}, t; \theta)$Ps​(u^∣I,t;θ). The overall update is an additive combination of gradients from comprehension and generation examples.

A central design choice is coupling comprehension and generation at the data level. For every positively rewarded interaction, the method converts data from one role into training data for the opposing role. Given the round $\rho$ρ datasets, the authors expand the comprehension set using successful generation data and the generation set using successful comprehension data:

Only positive rewards are converted. A successful listener prediction implies that the human utterance is a valid description for the target image, so it can train the speaker. Conversely, a successful speaker utterance provides a reliable comprehension target. Negative feedback is not converted because a failed generation may be due to ambiguity, a human error, or an incorrect listener choice, and using such examples for the opposite role can introduce misleading supervision. This data sharing procedure also introduces human utterances into the speaker’s training signal, helping to prevent the model’s language from drifting away from human language. The full continual learning loop and coupling mechanisms are shown in the framework diagram below:

Inference is also coupled between the two roles. For comprehension, the method defines a joint distribution that combines the listener distribution with the speaker distribution using a weighted geometric mean:

The hyperparameter $\lambda_l$λl​ controls how strongly the speaker model influences listener reasoning. Since the number of candidate targets is small, this joint distribution can be normalized exactly.

For generation, the method uses an analogous joint distribution:

Exact normalization is intractable because the space of possible utterances is large. Instead, the method samples $k$k candidate utterances from $P_s(u \mid \mathcal{I}, t; \theta)$Ps​(u∣I,t;θ) and normalizes the weighted geometric scores over those samples. In practice, the speaker term can overly favor short utterances during joint generation, so the authors set $\lambda_s = 0$λs​=0 for the speaker inference. Even then, the speaker model remains central because it supplies the candidate utterances that are subsequently re ranked by the listener model. This inference procedure has the character of a one level pragmatic reasoning process, where each role is evaluated partly through the perspective of the other role.

Experiment

The experiments use the KILOGRAM tangram reference game with an instruction-tuned vision-language model interacting with human listeners across multiple continual learning rounds, comparing a fully coupled system that shares data across comprehension and generation with ablations and a non-coupled baseline. The performance results show that coupling leads to faster and stronger improvement, especially for generation, and greatly improves sample efficiency, while human adaptation alone cannot explain the gains. The language analysis further indicates that coupling keeps utterances closer to human patterns, limits drift and vocabulary shrinkage, and preserves more context-dependent pragmatic diversity, although spatial reasoning and lexical richness remain limitations.