Command Palette
Search for a command to run...
Wissen, wann man nicht wiederverwenden sollte: Bedingter Erfahrungstransfer im autonomen Post-Training von LLMs
Wissen, wann man nicht wiederverwenden sollte: Bedingter Erfahrungstransfer im autonomen Post-Training von LLMs
Tingyun Li Wenfeng Feng Weiqing Li Abudukelimu Wuerkaixi Guohua Liu Yuewei Zhang
Zusammenfassung
Große Sprachmodelle bieten breite Fähigkeiten, doch ihre Anpassung an sich wandelnde Domänen, Werkzeuge und Anforderungen erfordert oft wiederholtes Post-Training. Autonome Systeme automatisieren Teile dieses Prozesses, indem sie Aktualisierungen vorschlagen, Kandidaten trainieren und Evaluierungsfeedback zur Auswahl nachfolgender Vorschläge nutzen. Mit zunehmender Evidenz tritt ein zentrales Problem zutage: Welche vergangenen Aktualisierungsnachweise bleiben handlungsrelevant, nachdem nachfolgendes Training das Ausgangsmodell verändert hat? Die Wirkung einer Aktualisierung hängt von ihrem Ausgangsmodell, den Daten und der Trainingsphase ab. Vergangenen Erfolg als kontextfreie Erlaubnis zu behandeln, kann Rechenressourcen verschwenden. Wird das resultierende Modell übernommen, kann dies zudem die nachfolgende Trainingsentwicklung beeinträchtigen. Wir formulieren dieses Problem als bedingten Erfahrungstransfer und stellen Boundary-Calibrated Intervention Transfer (BCIT) vor, eine Methode, die die Wiederverwendung von Erfahrungen vor gewichtsveränderndem Training autorisiert. BCIT bindet eine beobachtete Wirkung an ihren Quellkontext, prüft Anwendbarkeitsbedingungen, verwirft Kandidaten mit benannten harten Konflikten und erhebt bei Bedarf durch einen begrenzten Trainingsversuch Evidenz zum aktuellen Zustand. Vollständig trainierte Kandidaten unterliegen weiterhin einer gemeinsamen Übernahmeregel, und nur beobachtete Ereignisse erweitern das Gedächtnis. An einem 4B-Modell, das in den Bereichen Finanzlogik, Text-zu-SQL und Funktionsaufrufe angepasst wurde, zeigen Kandidatenaktualisierungen heterogene Zielund Beibehaltungseffekte über die evaluierten Kontexte hinweg. Bei gleichen Kandidaten, gleicher Evidenz und gleichem Rechenbudget autorisiert BCIT weniger schädliche Aktualisierungen und erreicht eine höhere Endmodellqualität bei gleichem Budget als die evaluierten Alternativen. Diese Ergebnisse stützen die Auffassung, dass die Autorisierung von Erfahrungen ein eigenständiges Problem im autonomen Post-Training darstellt.
One-sentence Summary
Alibaba Cloud Computing proposes Boundary-Calibrated Intervention Transfer (BCIT), a conditional experience transfer method for autonomous LLM post-training that verifies reuse conditions, vetoes hard conflicts, and obtains current-state evidence through bounded trials, thereby reducing harmful updates and improving final model quality on a 4B model across finance reasoning, text-to-SQL, and function calling tasks.
Key Contributions
- The paper formulates conditional experience transfer as a distinct problem in autonomous LLM post-training, where past update successes are not universally actionable after the parent model has changed.
- It introduces Boundary-Calibrated Intervention Transfer (BCIT), a method that binds observed effects to source contexts, checks applicability conditions, vetoes hard conflicts, and uses bounded training trials to gather current-state evidence before authorizing full training.
- Experiments on a 4B model adapted across finance reasoning, text-to-SQL, and function calling show that BCIT authorizes fewer harmful updates and achieves higher equal-budget final-model quality than evaluated alternatives, supporting experience-conditioned compute allocation.
Introduction
In many continual learning and multi-task settings, models are updated sequentially with post-training compute, but improvements from one context can inadvertently harm performance on others. Prior approaches often allocate compute unconditionally, assuming past success justifies further modification, which risks introducing conflicts and degrading overall capability. The authors introduce BCIT, a method that conditions post-training compute allocation on evidence from the current context. It rejects hard conflicts, validates candidate updates against the parent model, and applies a frozen rule to grant full training budgets, then uses a policy-shared adoption rule to promote or roll back each child model. This evidence-conditioned strategy authorizes fewer harmful updates while preserving beneficial ones, yielding higher cross-task performance than shared-model alternatives.
Experiment
The BCIT policy achieves the strongest trade-off, authorizing far fewer harmful outcomes than any other policy while still preserving most beneficial outcomes. Removing candidate-specific applicability from BCIT causes its harmful authorization rate to triple, reverting to the level of the source-evidence-only baseline. Policies like additive+veto also reduce harmful authorizations but sacrifice more beneficial outcomes than BCIT. BCIT authorizes only a quarter of harmful outcomes while retaining 90% of beneficial ones, yielding the lowest harmful share. Without candidate-specific applicability, BCIT's harmful authorization rate jumps to 75%, matching the source-evidence-only policy. Source-evidence-only keeps 90% of beneficial outcomes but still authorizes three-quarters of harmful outcomes. Flat-additive and applicability-only policies both keep 80% of beneficial outcomes and authorize nearly two-thirds of harmful outcomes.
Under a fixed 36 GPU-hour budget, the Add. + veto method achieves the highest cross-task average and the largest minimum improvement over the base model, while also delivering the best instruction-following scores among shared models. Single-task specialists surpass shared models on their respective tasks but consume the full budget per task, making them computationally expensive. The multi-task approaches consistently improve over the base model across all tasks without exceeding the budget. Add. + veto yields the highest cross-task mean (46.1) and the largest minimum gain over the base model (+2.7 points) among shared-model methods. Single-task specialists reach 39.6 on TAT-QA, 49.3 on BIRD, and 65.1 on BFCL, outperforming the best shared model but requiring 36 GPU-hours each. Instruction-following (IFEval) scores are highest for Add. + veto (76.0 prompt-level, 82.1 instruction-level) compared to other shared models.
In safety evaluations, the BCIT policy achieves the best trade-off between harmful and beneficial authorizations, drastically reducing harmful outcomes while preserving most beneficial ones; this advantage depends on candidate-specific applicability, without which harmful authorizations rise sharply. In multi-task training under a fixed compute budget, the Add. + veto method yields the highest cross-task average and instruction-following scores among shared models, while single-task specialists achieve higher per-task performance but at much greater computational cost.