Command Palette
Search for a command to run...
재사용하지 말아야 할 때를 아는 법: 자율적 LLM 사후 학습에서의 조건부 경험 전이
재사용하지 말아야 할 때를 아는 법: 자율적 LLM 사후 학습에서의 조건부 경험 전이
Tingyun Li Wenfeng Feng Weiqing Li Abudukelimu Wuerkaixi Guohua Liu Yuewei Zhang
초록
대규모 언어 모델은 광범위한 능력을 제공하지만, 진화하는 도메인, 도구 및 요구 사항에 적응시키려면 반복적인 사후 학습이 수반되는 경우가 많다. 자율 시스템은 업데이트를 제안하고, 후보 모델을 학습시키며, 평가 피드백을 사용하여 후속 제안을 선택함으로써 이 과정의 일부를 자동화한다. 증거가 축적됨에 따라 핵심적인 문제가 대두된다. 즉, 후속 학습으로 상위 모델이 변경된 후에도 과거의 어떤 업데이트 증거가 여전히 실행 가능한가 하는 점이다. 업데이트의 효과는 상위 모델, 데이터 및 학습 단계에 따라 달라진다. 과거의 성공을 맥락과 무관한 허가로 간주하면 연산 자원을 낭비할 수 있다. 그 결과 생성된 하위 모델이 채택될 경우, 후속 학습 궤적의 질을 저하시킬 수도 있다. 우리는 이 문제를 조건부 경험 전이로 정식화하고, 가중치를 변경하는 학습 전에 경험 재사용을 승인하는 방법인 경계 보정 개입 전이(BCIT)를 소개한다. BCIT는 관찰된 효과를 해당 원본 맥락에 결부시키고, 적용 조건을 확인하며, 명명된 하드 충돌이 있는 후보를 거부하고, 필요할 경우 제한된 학습 시도를 통해 현재 상태의 증거를 획득한다. 완전히 학습된 후보라도 공유된 채택 규칙을 적용받으며, 관찰된 이벤트만이 메모리를 확장한다. 금융 추론, 텍스트-SQL 변환 및 함수 호출 전반에 걸쳐 적응된 하나의 4B 모델에서, 후보 업데이트는 평가된 맥락 전반에 걸쳐 이질적인 목표 및 유지 효과를 나타냈다. 후보, 증거 및 연산량을 일치시킨 조건에서, BCIT는 평가된 대안들보다 유해한 업데이트를 더 적게 승인하고 동일 예산 내에서 더 높은 최종 모델 품질을 달성했다. 이러한 결과는 자율적 사후 학습에서 경험 승인을 별개의 문제로 다루는 것이 타당함을 뒷받침한다.
One-sentence Summary
Alibaba Cloud Computing proposes Boundary-Calibrated Intervention Transfer (BCIT), a conditional experience transfer method for autonomous LLM post-training that verifies reuse conditions, vetoes hard conflicts, and obtains current-state evidence through bounded trials, thereby reducing harmful updates and improving final model quality on a 4B model across finance reasoning, text-to-SQL, and function calling tasks.
Key Contributions
- The paper formulates conditional experience transfer as a distinct problem in autonomous LLM post-training, where past update successes are not universally actionable after the parent model has changed.
- It introduces Boundary-Calibrated Intervention Transfer (BCIT), a method that binds observed effects to source contexts, checks applicability conditions, vetoes hard conflicts, and uses bounded training trials to gather current-state evidence before authorizing full training.
- Experiments on a 4B model adapted across finance reasoning, text-to-SQL, and function calling show that BCIT authorizes fewer harmful updates and achieves higher equal-budget final-model quality than evaluated alternatives, supporting experience-conditioned compute allocation.
Introduction
In many continual learning and multi-task settings, models are updated sequentially with post-training compute, but improvements from one context can inadvertently harm performance on others. Prior approaches often allocate compute unconditionally, assuming past success justifies further modification, which risks introducing conflicts and degrading overall capability. The authors introduce BCIT, a method that conditions post-training compute allocation on evidence from the current context. It rejects hard conflicts, validates candidate updates against the parent model, and applies a frozen rule to grant full training budgets, then uses a policy-shared adoption rule to promote or roll back each child model. This evidence-conditioned strategy authorizes fewer harmful updates while preserving beneficial ones, yielding higher cross-task performance than shared-model alternatives.
Experiment
The BCIT policy achieves the strongest trade-off, authorizing far fewer harmful outcomes than any other policy while still preserving most beneficial outcomes. Removing candidate-specific applicability from BCIT causes its harmful authorization rate to triple, reverting to the level of the source-evidence-only baseline. Policies like additive+veto also reduce harmful authorizations but sacrifice more beneficial outcomes than BCIT. BCIT authorizes only a quarter of harmful outcomes while retaining 90% of beneficial ones, yielding the lowest harmful share. Without candidate-specific applicability, BCIT's harmful authorization rate jumps to 75%, matching the source-evidence-only policy. Source-evidence-only keeps 90% of beneficial outcomes but still authorizes three-quarters of harmful outcomes. Flat-additive and applicability-only policies both keep 80% of beneficial outcomes and authorize nearly two-thirds of harmful outcomes.
Under a fixed 36 GPU-hour budget, the Add. + veto method achieves the highest cross-task average and the largest minimum improvement over the base model, while also delivering the best instruction-following scores among shared models. Single-task specialists surpass shared models on their respective tasks but consume the full budget per task, making them computationally expensive. The multi-task approaches consistently improve over the base model across all tasks without exceeding the budget. Add. + veto yields the highest cross-task mean (46.1) and the largest minimum gain over the base model (+2.7 points) among shared-model methods. Single-task specialists reach 39.6 on TAT-QA, 49.3 on BIRD, and 65.1 on BFCL, outperforming the best shared model but requiring 36 GPU-hours each. Instruction-following (IFEval) scores are highest for Add. + veto (76.0 prompt-level, 82.1 instruction-level) compared to other shared models.
In safety evaluations, the BCIT policy achieves the best trade-off between harmful and beneficial authorizations, drastically reducing harmful outcomes while preserving most beneficial ones; this advantage depends on candidate-specific applicability, without which harmful authorizations rise sharply. In multi-task training under a fixed compute budget, the Add. + veto method yields the highest cross-task average and instruction-following scores among shared models, while single-task specialists achieve higher per-task performance but at much greater computational cost.