Command Palette
Search for a command to run...
자기지도 강화학습을 위한 1000층 네트워크: 깊이 확장이 새로운 목표 도달 능력을 가능하게 한다
자기지도 강화학습을 위한 1000층 네트워크: 깊이 확장이 새로운 목표 도달 능력을 가능하게 한다
Kevin Wang Ishaan Javali Michał Bortkiewicz Tomasz Trzciński Benjamin Eysenbach
초록
자기지도 학습의 규모 확장은 언어 및 시각 분야에서 혁신적 성과를 이룩하였으나, 강화학습(Reinforcement Learning, RL)에서 이에 상응하는 발전은 아직 달성되지 못하고 있다. 본 논문에서는 확장성에서 substantial한 개선을 가능하게 하는 자기지도 강화학습의 구성 요소를 연구하며, 특히 네트워크 깊이가 핵심 요인으로 작용함을 입증한다. 최근 대부분의 강화학습 논문은 얕은 아키텍처(약 2–5층)에 의존해 왔으나, 우리는 깊이를 최대 1024층까지 증가시킴으로써 성능을 현저히 향상시킬 수 있음을 보인다. 실험은 데모나 보상 없이 수행되는 무감독 목표 조건 설정(unsupervised goal-conditioned setting)에서 진행되며, 에이전트는 처음부터 탐색하며 명령된 목표에 도달할 확률을 극대화하는 방법을 학습해야 한다. 시뮬레이션 기반 보행 및 조작 과제에서 평가한 결과, 본 접근법은 자기지도 대조적 강화학습(self-supervised contrastive RL) 알고리즘의 성능을 2×–50× 향상시켜 다른 목표 조건 기반 기준선들을 상회한다. 모델 깊이의 증가는 성공률을 높일 뿐만 아니라 학습되는 행동의 질적 변화도 가져온다. 프로젝트 웹페이지 및 코드는 다음 링크에서 확인할 수 있다: https://wang-kevin3290.github.io/scaling-crl/.
One-sentence Summary
Princeton University and Warsaw University of Technology et al. demonstrate that scaling network depth to 1024 layers in self-supervised contrastive RL, far beyond the typical 2–5 layers, improves goal-reaching performance by 2×−50× on unsupervised locomotion and manipulation tasks while qualitatively changing the behaviors the agent learns.
Key Contributions
- The paper introduces building blocks for self-supervised reinforcement learning in an unsupervised goal-conditioned setting, where an agent explores from scratch and learns to maximize the likelihood of reaching commanded goals without demonstrations or rewards.
- The approach combines an InfoNCE-trained contrastive critic whose output is the l2 norm between state-action and goal embeddings, a policy trained to maximize this critic, and residual connections that help stabilize training of networks with up to 1024 layers.
- Simulated locomotion and manipulation experiments show that increasing depth improves success rates by 2x to 50x over self-supervised contrastive RL and other goal-conditioned baselines, while producing qualitatively richer goal-reaching behaviors.
Introduction
Scaling large models has produced emergent capabilities in vision and language, yet in reinforcement learning the payoff from scale remains uncertain because agents receive sparse reward signals and usually use small MLP policies of 2 to 5 layers. Prior work has struggled to match these gains, often seeing only marginal improvements, training instability, or limited and sometimes negative returns when increasing depth, while commonly treating self supervised pretraining as the main path to scale. The authors leverage contrastive reinforcement learning as a self supervised actor critic method, increase online data availability through GPU accelerated RL training, and build networks up to about 100 times deeper using residual connections, layer normalization, and Swish activations. They show that this combination enables strong RL scalability, with more than 20 times performance gains in half the environments, better results than goal conditioned baselines, and qualitatively distinct policies emerging as depth and scale increase.
Experiment
The experiments evaluate goal-conditioned contrastive RL on JaxGCRL locomotion, navigation, and manipulation tasks with sparse rewards, comparing residual actor and critic networks of increasing depth against standard shallow baselines. The results show that depth scaling yields substantial gains, especially in long-horizon and humanoid tasks, and leads to emergent navigational and locomotion skills once critical depths are reached. Ablations and mechanistic analyses indicate that depth is more useful than width, joint actor-critic scaling can help, and larger depths unlock larger batches while improving representation quality, exploration, and limited generalization, though offline results do not show the same benefit.