HyperAIHyperAI

Command Palette

Search for a command to run...

Multimodal

ForceHarness: Kraft in Aktion und Physik für eingefrorene Frontier-Modelle verwandeln

Ke Sun Kevin Yuchen Ma Yike Zhang Yanzhe Chen Mike Zheng Shou

Zusammenfassung

Frontier-multimodale Modelle können Roboter inzwischen per Zero-Shot anhand von Bildern steuern, doch viele Manipulationsaufgaben hängen von Kontaktkräften ab, die Bilder nicht zeigen können. Die Einbindung von Kraft in ein ausschließlich über eine API zugängliches, eingefrorenes Modell wirft zwei Herausforderungen auf. Erstens entwickelt sich der Kontakt viel schneller, als ein API-Modell entscheiden kann, sodass das System den Kraftregelkreis unterhalb des Modells schließen muss. Zweitens kann ein eingefrorenes Modell rohe Kraftverläufe nicht interpretieren, sodass die Kraft in einer Form zurückgegeben werden muss, über die das Modell nachdenken kann. Daher stellen wir ForceHarness vor, ein Rahmenwerk, das eingefrorenen Modellen die Nutzung von Kraft ohne Demonstrationen oder Feinabstimmung ermöglicht. Nach der Kalibrierung der Handkraft allein aus einem Handgelenkssensor oder Gelenkdrehmomenten adressiert ForceHarness die erste Herausforderung mit selbstterminierenden Kraftprimitiven, die die Kraft eigenständig regeln und bei einem Kontaktereignis zurückkehren. Zur Bewältigung der zweiten Herausforderung wandelt eine Physik-Übersetzungsschicht jeden Kraftverlauf in physikalische Größen mit Unsicherheit als Werkzeugergebnis um. Um ForceHarness unter identischen Bedingungen mit erfahrenen Kraftreglern und gelernten Strategien zu vergleichen, entwickeln wir außerdem HiddenPhys, einen MuJoCo-Benchmark mit sieben Aufgaben und verborgenen physikalischen Variablen. Auf einem realen Arm ohne Handgelenks-Kraftsensor erhöht ForceHarness die Zero-Shot-Erfolgsquote von GPT-6 Astra von 3,3 % auf 90,0 %. In der Simulation erzielt es einen Zuwachs von 17,5 Prozentpunkten auf der öffentlichen ForceBench-Suite und 34,3 Prozentpunkten auf HiddenPhys, übertrifft damit auf beiden Benchmarks kraftbewusste, auf Demonstrationen trainierte Strategien und erreicht auf HiddenPhys das Niveau handgeschriebener Expertenregler. Ablationsstudien zeigen zudem, dass kleinere Modelle deutlich bessere Entscheidungen treffen, wenn ihnen physikalische Größen statt roher Kraftwerte bereitgestellt werden.

One-sentence Summary

Researchers at Show Lab, National University of Singapore, propose ForceHarness, a harness that enables frozen frontier models to use force without demonstrations or finetuning via self-terminating force primitives and a physics translation layer that converts force traces into uncertainty-aware physical quantities, raising GPT-6 Astra zero-shot success from 3.3%3.3\%3.3% to 90.0%90.0\%90.0% and outperforming learned force-aware policies.

Key Contributions

  • ForceHarness is introduced as a three-layer harness that combines force calibration, self-terminating force primitives, and physics translation to give an API-only frozen multimodal model force-enabled manipulation without demonstrations or finetuning.
  • HiddenPhys is introduced as a seven-task MuJoCo benchmark with hidden physical variables and identical seeds, including expert force controllers and learned force policies for controlled comparison.
  • Experiments show ForceHarness raises GPT-6 Astra's zero-shot success rate on a real arm without a wrist force sensor from 3.3% to 90.0%, adds 17.5 percentage points on ForceBench and 34.3 percentage points on HiddenPhys, and matches hand-written expert controllers on HiddenPhys while surpassing force-aware policies trained on demonstrations on both benchmarks.

Introduction

Frontier multimodal models can already control robots zero-shot through frozen APIs, but existing harnesses mainly help them perceive and move rather than reason about physical contact. Many manipulation tasks, such as engaging a screw thread or comparing identical-looking bottles by weight, depend on force signals that images cannot provide. Prior force-aware approaches either require demonstrations and model weight updates, or expose raw sensor readings and low-level force-control code without calibrated physical feedback. The authors introduce ForceHarness, a three-layer system that combines force calibration, self-terminating force primitives, and physics translation so a frozen model can use wrist force without any task-specific demonstrations or training.

Dataset

The authors introduce HiddenPhys, a benchmark built on seven robosuite/MuJoCo manipulation tasks. It is designed to complement public force benchmarks such as ForceBench.

  • Sources and composition: HiddenPhys contains seven tasks based on real-world contact scenarios: Weight, CoM, Wipe, Stack, Peg, Latch, and Lock.
  • Hidden variable schema: Each task hides one physical variable that determines success. The instruction gives the range of the hidden variable, but the value is drawn from the episode seed, so all methods see the same episodes.
  • Observation and processing: A Panda arm uses operational space control. Force is the wrist sensor reading with added noise and a per-episode bias. To isolate force from visual perception, a frozen model reads object poses as noisy text instead of images.
  • Scale and references: Each task has a hand-written expert force controller and a position-only controller that moves without force. Learned references include Diffusion Policy and π0.5, each trained with and without force input, plus a multi-task ForceVLA variant. Learned policies train on about 200 expert demonstrations per task. No separate validation split or mixture ratio is specified in this excerpt.
  • Usage: The benchmark is used to train and compare force-aware policies, reporting both success and damage. Damage is counted when the force on a handle exceeds a task-specific damage threshold.

Method

The authors design ForceHarness, a middleware framework that bridges a frozen frontier model and a robotic manipulator. Because the model operates via an API and requires several seconds for each decision, the system processes force feedback at two distinct rates: high-frequency contact control within primitives and low-frequency decision-making via tool results. The architecture comprises three distinct layers, as shown in the figure below:

The bottom layer handles force sensing and calibration, providing a calibrated estimate of the environmental force FFF applied to the robot hand. When a wrist force/torque sensor is unavailable, the system recovers the force from the external joint torques τext\tau_{\mathrm{ext}}τext​ estimated by a torque-controlled arm. It solves for the wrench www at the flange using least squares and then removes a zero-point bias bbb to compute the force:

w=(J⊤)+τext,F=−(wF−b)w = (J^\top)^+ \tau_{\mathrm{ext}}, \qquad F = -(w_F - b)w=(J⊤)+τext​,F=−(wF​−b)

where JJJ is the flange Jacobian and wFw_FwF​ is the force component of www. To mitigate the slow drift of the zero point, the layer controls when readings are taken rather than relying on filtering. It tares at rest, re-zeroing bbb before each empty-hand primitive, and averages the force over a short window after the arm settles. The system reports force changes relative to a reference from the same episode, ensuring the model compares relative readings instead of trusting absolute values.

The middle layer consists of a menu of self-terminating force primitives. These primitives package contact skills into tools that the model calls with parameters such as movement direction and target force. Since most robotic arms accept position targets rather than direct force commands, the primitives leverage the stock impedance controller, which acts as a spring between the hand and the target pose. A primitive applies a force −F-F−F by allowing a target pose p∗p^*p∗ to lead the measured pose ppp:

−F≈K(p∗−p),∥p∗−p∥≤L-F \approx K(p^* - p), \qquad \|p^* - p\| \leq L−F≈K(p∗−p),∥p∗−p∥≤L

where KKK is the controller stiffness and LLL bounds the maximum force. All primitives share a single control loop. To hold a target force FdF_dFd​ along a direction uuu, the control law converts the force error into a target velocity using admittance control:

v=clip(kf(Fd−Fu),±vmax⁡),Δp∗=uvΔtv = \mathrm{clip}\big(k_f(F_d - F_u), \pm v_{\max}\big), \quad \Delta p^* = u v \Delta tv=clip(kf​(Fd​−Fu​),±vmax​),Δp∗=uvΔt

where FuF_uFu​ is the reaction force against uuu. Crucially, these primitives are self-terminating. Instead of running on a fixed clock, a primitive stops immediately upon detecting a contact event, meeting its goal, or hitting a safety cap. This division of labor allows the slow-deciding model to delegate precise timing and low-level control to the primitives.

The top layer is the physics translation layer, which converts the raw force trace logged by a primitive into interpretable physical quantities with propagated uncertainties. Since the frozen model lacks the ability to parse raw force data, this layer uses simple, fixed estimators based on averages and least-squares fits. For example, payload mass mmm and friction coefficient μ\muμ are derived as:

m=∣Fˉz∣/g,σm=sz2/n+σb2/gm = |\bar{F}_z| / g, \quad \sigma_m = \sqrt{s_z^2 / n + \sigma_b^2} / gm=∣Fˉz​∣/g,σm​=sz2​/n+σb2​​/g μ=∣Ft∣‾/Fˉn,(σμ/μ)2=(σt/∣Ft∣‾)2+(σn/Fˉn)2\mu = \overline{|F_t|} / \bar{F}_n, \quad (\sigma_\mu / \mu)^2 = (\sigma_t / \overline{|F_t|})^2 + (\sigma_n / \bar{F}_n)^2μ=∣Ft​∣​/Fˉn​,(σμ​/μ)2=(σt​/∣Ft​∣​)2+(σn​/Fˉn​)2

where szs_zsz​ is the sample standard deviation of the vertical force, and σt\sigma_tσt​ and σn\sigma_nσn​ are the standard errors of the tangential and normal forces. The layer strips away the zero bias and held loads, isolating the specific window where a quantity holds a single physical meaning. It returns these values alongside uncertainty σ\sigmaσ and status flags, deliberately omitting task-specific thresholds so that the model retains full judgment over the next action.

In the operational loop, the model reads the instruction, tool menu, call history, and the latest translated observations to output a single tool call. Once a primitive executes and terminates, its translated physical quantities are returned as the tool result. The model then evaluates the outcome, such as determining if a part has successfully seated, and selects the subsequent tool call until the episode concludes. Throughout this process, a fixed guard beneath the primitives enforces strict limits on force, speed, and workspace, ensuring safety regardless of the model's commands.

Experiment

The evaluation spans a real Franka FR3 without a wrist force sensor, the public ForceBench suite, and the HiddenPhys benchmark, comparing ForceHarness against a no-force control and trained policies. Real-robot experiments show that force feedback enables reliable contact manipulation and lets the model infer invisible properties such as mass and stiffness, while ForceBench confirms gains mainly when subtle resistance determines success. HiddenPhys results show ForceHarness matching a hand-written expert with safe force handling and remaining robust under distribution shift, whereas learned policies struggle and often damage handles. Ablations attribute the benefits to self-terminating force primitives for safe contact, harness-computed physical quantities for smaller models, and turn-by-turn feedback for adaptation after failures.

The twelve force primitives organize contact skills into five groups, covering approach and contact, holding force, searching and seating, turning under load, and measurement. They return concise physical quantities rather than raw sensor traces, helping language models make safer and more effective contact decisions. Experiments show that these primitives improve task success, reduce hardware damage, and lower cost across models and tasks. The primitives apply force by letting a target pose lead the measured pose under a clamp, which bounds force and leaves low-level robot control unchanged. Self-terminating primitives alone improve contact task success and sharply reduce handle damage, even without force trace feedback. Returning harness-computed physical quantities rather than raw samples or force plots improves decision accuracy and success for smaller language models. The force interface substantially improves one-shot program performance, and turn-by-turn feedback adds further gains by adapting after failures.

HiddenPhys is a seven-task manipulation benchmark where each task hides one physical variable that determines success, such as object mass, center of mass, surface height, or mechanism stiffness. The tasks require force sensing to measure held loads, apply bounded contact forces, search by touch, and avoid damage. Hidden values are drawn per episode from stated ranges, and the benchmark pairs these tasks with reference controllers and learned policies. The seven tasks split into judging held loads, finding uncertain surfaces, and operating mechanisms by touch. Force requirements range from measuring load properties and pressing within a safe force band to feeling for contact or a drop and limiting handle force below thresholds.

Adding ForceHarness dramatically improves real-robot success over a no-force baseline. The no-force baseline struggles on contact tasks and fails completely on bottle sorting and soft versus hard discrimination, while ForceHarness achieves high success across all tasks and transfers to a second model. Across all six tasks, ForceHarness lifts GPT-6 Astra from 2 of 60 to 54 of 60 successful episodes. Without force, the model scores zero on sorting bottles by weight and telling soft from hard; with ForceHarness, both tested models solve these non-visual tasks at or near ceiling.

Zero-shot models with ForceHarness outperform the best listed trained ForceBench policy by a wide margin, with Astra and Sol reaching 87.5% and Fable reaching 75.0% versus 35.31% for the strongest trained baseline. No-force zero-shot baselines already exceed that trained baseline, so model capability, harness, and instructions contribute beyond force feedback. Within the harness, force feedback produces paired gains mainly on Drawer Pull, where success improves from zero to perfect. Zero-shot models with ForceHarness surpass the strongest listed trained ForceBench baseline by a large margin. Force feedback primarily improves Drawer Pull, raising it from 0/10 to 10/10, while other tasks do not separate at 10 episodes. No-force zero-shot baselines already outperform the best listed trained baseline, indicating force is not the only factor.

On HiddenPhys, force-aware models with ForceHarness reach near-expert success and avoid handle damage, closely matching a hand-written force controller. Learned and position-only policies score lower and frequently damage handles, while adding passive wrist force to learned policies does not consistently improve success or safety. Under shifted hidden ranges, force-aware models degrade less than no-force and learned baselines and track the fixed-threshold expert. ForceHarness raises model success to near expert level and eliminates handle damage, while learned policies remain lower and show frequent damage. Adding wrist force to learned policies brings no consistent gain, suggesting passive force conditioning alone does not teach active probing or safe force limiting. Under distribution shift, ForceHarness models show smaller relative drops than no-force baselines and closely match the fixed-threshold expert.

Across real-robot and benchmark evaluations, the ForceHarness interface with twelve self-terminating force primitives is tested on contact-heavy manipulation tasks, HiddenPhys tasks with hidden physical variables, and zero-shot model comparisons against trained policies. The experiments show that returning compact physical quantities from the harness improves task success, reduces hardware damage, and enables non-visual capabilities such as weight sorting or soft-versus-hard discrimination. Zero-shot models using the harness outperform trained baselines, with force feedback mainly helping tasks like drawer pulling, while passive wrist force alone does not provide consistent safety or success gains. On HiddenPhys, force-aware models match expert-level success, avoid handle damage, and generalize better under shifted conditions than no-force or learned policies.


KI mit KI entwickeln

Von der Idee bis zum Launch – beschleunigen Sie Ihre KI-Entwicklung mit kostenlosem KI-Co-Coding, sofort einsatzbereiter Umgebung und bestem GPU-Preis.

KI-gestütztes kollaboratives Programmieren
Sofort einsatzbereite GPUs
Die besten Preise

HyperAI Newsletters

Abonnieren Sie unsere neuesten Updates
Wir werden die neuesten Updates der Woche in Ihren Posteingang liefern um neun Uhr jeden Montagmorgen
Unterstützt von MailChimp