Command Palette
Search for a command to run...
ForceHarness: 동결된 프런티어 모델을 위해 힘을 행동과 물리로 전환하기
ForceHarness: 동결된 프런티어 모델을 위해 힘을 행동과 물리로 전환하기
Ke Sun Kevin Yuchen Ma Yike Zhang Yanzhe Chen Mike Zheng Shou
초록
프런티어 멀티모달 모델은 이제 이미지로부터 제로샷으로 로봇을 제어할 수 있지만, 많은 조작 과제는 이미지로는 드러나지 않는 접촉력에 의존한다. API 전용 동결 모델에 힘을 도입하는 데는 두 가지 어려움이 있다. 첫째, 접촉은 API 모델이 결정을 내릴 수 있는 속도보다 훨씬 빠르게 변화하므로, 시스템은 모델 아래 계층에서 힘에 대한 폐루프 제어를 수행해야 한다. 둘째, 동결 모델은 원시 힘 스트림을 해석할 수 없으므로, 힘은 모델이 추론할 수 있는 형태로 반환되어야 한다. 이에 본 연구는 시연이나 파인튜닝 없이 동결 모델이 힘을 사용할 수 있게 하는 하네스인 ForceHarness를 제시한다. 손목 센서 또는 관절 토크만으로 손 힘을 보정한 뒤, ForceHarness는 첫 번째 어려움을 스스로 힘을 조절하고 접촉 이벤트에서 반환하는 자기 종료형 힘 프리미티브로 해결한다. 두 번째 어려움을 해결하기 위해, 물리 변환 계층이 각 힘 궤적을 불확실성을 포함한 물리량으로 변환하여 도구 결과로 제공한다. 동일한 조건에서 ForceHarness를 전문가 힘 제어기 및 학습된 정책과 비교 평가하기 위해, 숨겨진 물리 변수를 포함하는 7개 과제 MuJoCo 벤치마크인 HiddenPhys도 구축한다. 손목 힘 센서가 없는 실제 로봇 팔에서 ForceHarness는 GPT-6 Astra의 제로샷 성공률을 3.3%에서 90.0%로 향상시킨다. 시뮬레이션에서는 공개 ForceBench 스위트에서 17.5%포인트, HiddenPhys에서 34.3%포인트의 성능 향상을 보이며, 두 벤치마크 모두에서 시연으로 학습된 힘 인지 정책을 능가하고 HiddenPhys에서는 수작업 전문가 제어기와 동등한 성능을 달성한다. 추가 절제 실험은 더 작은 모델이 원시 힘 값보다 물리량을 제공받을 때 유의미하게 더 나은 결정을 내린다는 것을 보여준다.
One-sentence Summary
Researchers at Show Lab, National University of Singapore, propose ForceHarness, a harness that enables frozen frontier models to use force without demonstrations or finetuning via self-terminating force primitives and a physics translation layer that converts force traces into uncertainty-aware physical quantities, raising GPT-6 Astra zero-shot success from 3.3% to 90.0% and outperforming learned force-aware policies.
Key Contributions
- ForceHarness is introduced as a three-layer harness that combines force calibration, self-terminating force primitives, and physics translation to give an API-only frozen multimodal model force-enabled manipulation without demonstrations or finetuning.
- HiddenPhys is introduced as a seven-task MuJoCo benchmark with hidden physical variables and identical seeds, including expert force controllers and learned force policies for controlled comparison.
- Experiments show ForceHarness raises GPT-6 Astra's zero-shot success rate on a real arm without a wrist force sensor from 3.3% to 90.0%, adds 17.5 percentage points on ForceBench and 34.3 percentage points on HiddenPhys, and matches hand-written expert controllers on HiddenPhys while surpassing force-aware policies trained on demonstrations on both benchmarks.
Introduction
Frontier multimodal models can already control robots zero-shot through frozen APIs, but existing harnesses mainly help them perceive and move rather than reason about physical contact. Many manipulation tasks, such as engaging a screw thread or comparing identical-looking bottles by weight, depend on force signals that images cannot provide. Prior force-aware approaches either require demonstrations and model weight updates, or expose raw sensor readings and low-level force-control code without calibrated physical feedback. The authors introduce ForceHarness, a three-layer system that combines force calibration, self-terminating force primitives, and physics translation so a frozen model can use wrist force without any task-specific demonstrations or training.
Dataset
The authors introduce HiddenPhys, a benchmark built on seven robosuite/MuJoCo manipulation tasks. It is designed to complement public force benchmarks such as ForceBench.
- Sources and composition: HiddenPhys contains seven tasks based on real-world contact scenarios: Weight, CoM, Wipe, Stack, Peg, Latch, and Lock.
- Hidden variable schema: Each task hides one physical variable that determines success. The instruction gives the range of the hidden variable, but the value is drawn from the episode seed, so all methods see the same episodes.
- Observation and processing: A Panda arm uses operational space control. Force is the wrist sensor reading with added noise and a per-episode bias. To isolate force from visual perception, a frozen model reads object poses as noisy text instead of images.
- Scale and references: Each task has a hand-written expert force controller and a position-only controller that moves without force. Learned references include Diffusion Policy and π0.5, each trained with and without force input, plus a multi-task ForceVLA variant. Learned policies train on about 200 expert demonstrations per task. No separate validation split or mixture ratio is specified in this excerpt.
- Usage: The benchmark is used to train and compare force-aware policies, reporting both success and damage. Damage is counted when the force on a handle exceeds a task-specific damage threshold.
Method
The authors design ForceHarness, a middleware framework that bridges a frozen frontier model and a robotic manipulator. Because the model operates via an API and requires several seconds for each decision, the system processes force feedback at two distinct rates: high-frequency contact control within primitives and low-frequency decision-making via tool results. The architecture comprises three distinct layers, as shown in the figure below:
The bottom layer handles force sensing and calibration, providing a calibrated estimate of the environmental force F applied to the robot hand. When a wrist force/torque sensor is unavailable, the system recovers the force from the external joint torques τext estimated by a torque-controlled arm. It solves for the wrench w at the flange using least squares and then removes a zero-point bias b to compute the force:
w=(J⊤)+τext,F=−(wF−b)where J is the flange Jacobian and wF is the force component of w. To mitigate the slow drift of the zero point, the layer controls when readings are taken rather than relying on filtering. It tares at rest, re-zeroing b before each empty-hand primitive, and averages the force over a short window after the arm settles. The system reports force changes relative to a reference from the same episode, ensuring the model compares relative readings instead of trusting absolute values.
The middle layer consists of a menu of self-terminating force primitives. These primitives package contact skills into tools that the model calls with parameters such as movement direction and target force. Since most robotic arms accept position targets rather than direct force commands, the primitives leverage the stock impedance controller, which acts as a spring between the hand and the target pose. A primitive applies a force −F by allowing a target pose p∗ to lead the measured pose p:
−F≈K(p∗−p),∥p∗−p∥≤Lwhere K is the controller stiffness and L bounds the maximum force. All primitives share a single control loop. To hold a target force Fd along a direction u, the control law converts the force error into a target velocity using admittance control:
v=clip(kf(Fd−Fu),±vmax),Δp∗=uvΔtwhere Fu is the reaction force against u. Crucially, these primitives are self-terminating. Instead of running on a fixed clock, a primitive stops immediately upon detecting a contact event, meeting its goal, or hitting a safety cap. This division of labor allows the slow-deciding model to delegate precise timing and low-level control to the primitives.
The top layer is the physics translation layer, which converts the raw force trace logged by a primitive into interpretable physical quantities with propagated uncertainties. Since the frozen model lacks the ability to parse raw force data, this layer uses simple, fixed estimators based on averages and least-squares fits. For example, payload mass m and friction coefficient μ are derived as:
m=∣Fˉz∣/g,σm=sz2/n+σb2/g μ=∣Ft∣/Fˉn,(σμ/μ)2=(σt/∣Ft∣)2+(σn/Fˉn)2where sz is the sample standard deviation of the vertical force, and σt and σn are the standard errors of the tangential and normal forces. The layer strips away the zero bias and held loads, isolating the specific window where a quantity holds a single physical meaning. It returns these values alongside uncertainty σ and status flags, deliberately omitting task-specific thresholds so that the model retains full judgment over the next action.
In the operational loop, the model reads the instruction, tool menu, call history, and the latest translated observations to output a single tool call. Once a primitive executes and terminates, its translated physical quantities are returned as the tool result. The model then evaluates the outcome, such as determining if a part has successfully seated, and selects the subsequent tool call until the episode concludes. Throughout this process, a fixed guard beneath the primitives enforces strict limits on force, speed, and workspace, ensuring safety regardless of the model's commands.
Experiment
The evaluation spans a real Franka FR3 without a wrist force sensor, the public ForceBench suite, and the HiddenPhys benchmark, comparing ForceHarness against a no-force control and trained policies. Real-robot experiments show that force feedback enables reliable contact manipulation and lets the model infer invisible properties such as mass and stiffness, while ForceBench confirms gains mainly when subtle resistance determines success. HiddenPhys results show ForceHarness matching a hand-written expert with safe force handling and remaining robust under distribution shift, whereas learned policies struggle and often damage handles. Ablations attribute the benefits to self-terminating force primitives for safe contact, harness-computed physical quantities for smaller models, and turn-by-turn feedback for adaptation after failures.
The twelve force primitives organize contact skills into five groups, covering approach and contact, holding force, searching and seating, turning under load, and measurement. They return concise physical quantities rather than raw sensor traces, helping language models make safer and more effective contact decisions. Experiments show that these primitives improve task success, reduce hardware damage, and lower cost across models and tasks. The primitives apply force by letting a target pose lead the measured pose under a clamp, which bounds force and leaves low-level robot control unchanged. Self-terminating primitives alone improve contact task success and sharply reduce handle damage, even without force trace feedback. Returning harness-computed physical quantities rather than raw samples or force plots improves decision accuracy and success for smaller language models. The force interface substantially improves one-shot program performance, and turn-by-turn feedback adds further gains by adapting after failures.
HiddenPhys is a seven-task manipulation benchmark where each task hides one physical variable that determines success, such as object mass, center of mass, surface height, or mechanism stiffness. The tasks require force sensing to measure held loads, apply bounded contact forces, search by touch, and avoid damage. Hidden values are drawn per episode from stated ranges, and the benchmark pairs these tasks with reference controllers and learned policies. The seven tasks split into judging held loads, finding uncertain surfaces, and operating mechanisms by touch. Force requirements range from measuring load properties and pressing within a safe force band to feeling for contact or a drop and limiting handle force below thresholds.
Adding ForceHarness dramatically improves real-robot success over a no-force baseline. The no-force baseline struggles on contact tasks and fails completely on bottle sorting and soft versus hard discrimination, while ForceHarness achieves high success across all tasks and transfers to a second model. Across all six tasks, ForceHarness lifts GPT-6 Astra from 2 of 60 to 54 of 60 successful episodes. Without force, the model scores zero on sorting bottles by weight and telling soft from hard; with ForceHarness, both tested models solve these non-visual tasks at or near ceiling.
Zero-shot models with ForceHarness outperform the best listed trained ForceBench policy by a wide margin, with Astra and Sol reaching 87.5% and Fable reaching 75.0% versus 35.31% for the strongest trained baseline. No-force zero-shot baselines already exceed that trained baseline, so model capability, harness, and instructions contribute beyond force feedback. Within the harness, force feedback produces paired gains mainly on Drawer Pull, where success improves from zero to perfect. Zero-shot models with ForceHarness surpass the strongest listed trained ForceBench baseline by a large margin. Force feedback primarily improves Drawer Pull, raising it from 0/10 to 10/10, while other tasks do not separate at 10 episodes. No-force zero-shot baselines already outperform the best listed trained baseline, indicating force is not the only factor.
On HiddenPhys, force-aware models with ForceHarness reach near-expert success and avoid handle damage, closely matching a hand-written force controller. Learned and position-only policies score lower and frequently damage handles, while adding passive wrist force to learned policies does not consistently improve success or safety. Under shifted hidden ranges, force-aware models degrade less than no-force and learned baselines and track the fixed-threshold expert. ForceHarness raises model success to near expert level and eliminates handle damage, while learned policies remain lower and show frequent damage. Adding wrist force to learned policies brings no consistent gain, suggesting passive force conditioning alone does not teach active probing or safe force limiting. Under distribution shift, ForceHarness models show smaller relative drops than no-force baselines and closely match the fixed-threshold expert.
Across real-robot and benchmark evaluations, the ForceHarness interface with twelve self-terminating force primitives is tested on contact-heavy manipulation tasks, HiddenPhys tasks with hidden physical variables, and zero-shot model comparisons against trained policies. The experiments show that returning compact physical quantities from the harness improves task success, reduces hardware damage, and enables non-visual capabilities such as weight sorting or soft-versus-hard discrimination. Zero-shot models using the harness outperform trained baselines, with force feedback mainly helping tasks like drawer pulling, while passive wrist force alone does not provide consistent safety or success gains. On HiddenPhys, force-aware models match expert-level success, avoid handle damage, and generalize better under shifted conditions than no-force or learned policies.