One-sentence Summary

Researchers at Robbyant develop LingBot-Touch, a tactile foundation model that learns unified representations across piezoresistive, magnetic, and visuotactile sensors and provides plug-and-play tactile tokens for diffusion and vision-language-action policies such as $\pi$π and LingBot-VLA, improving manipulation dexterity in both simulation and real-robot tasks.

Key Contributions

• LingBot-Touch combines a multi-branch tokenizer with a shared Haptic Transformer to learn unified contact representations across piezoresistive, magnetic, and visuotactile sensors.
• A canonical fingertip space and Tactile-JEPA objective train the shared encoder by predicting contact in latent space, yielding plug-and-play tactile tokens for diffusion policies and vision-language-action models such as π and LingBot-VLA.
• Frozen after pretraining on 5,000 hours of tactile recordings, the model improves imitation learning, VLA, and reinforcement learning policies across 26 real-world and 12 simulated tasks, transfers to an unseen magnetic hand, outperforms FTP-1 with a sensor-specific adapter, and maintains strong computational efficiency.

Introduction

Humans depend on touch for dexterous manipulation, and robots still manipulate mostly by sight even though vision cannot reveal contact force, slip, or what happens under an occluding finger. Touch has not scaled like vision or language because there is no common input format across piezoresistive, magnetic, and optical sensors, no large pooled corpus, and no shared representation or established learning recipe. Prior tactile learning remains fragmented across sensor-specific studies and individual tasks. The authors introduce LingBot-Touch, a tactile foundation model that learns a shared contact representation from all three dominant sensing principles with one encoder and extends to new sensors without retraining. It does so by mapping sensor taxels into a canonical fingertip space, using a latent predictive objective suited to sparse unlabeled tactile signals, and training on a 5,000-hour corpus from human tactile gloves and teleoperated robot hands.

Dataset

The authors build a multi-source tactile dataset with 4,983 hours of recordings, 464,431 interaction episodes, and approximately 1.33 billion frames.

• Sources and composition:

  • Human operators wearing tactile gloves provide the bulk of the corpus. These recordings use piezoresistive and magnetic gloves and cover everyday manipulation in ordinary environments.
  • Teleoperation provides the remainder, using a dual-arm system with Sharpa Wave dexterous hands whose fingertips carry visuotactile sensors. This source records the same task families as the robot hand used later for deployment.
  • Glove recordings account for about two thirds of the corpus; teleoperation accounts for about one third.

• Subset details:

  • Piezoresistive glove subset: recorded from human operators, represented within the glove portion of the corpus.
  • Magnetic glove subset: recorded from human operators, also represented within the glove portion.
  • Visuotactile teleoperation subset: recorded from Sharpa Wave fingertips, represented within the teleoperation portion.
  • The three sensor sources differ in transduction principle, taxels per finger, sampling rate, and fingertip geometry.

• Behavioral labeling and metadata:

  • Every episode is assigned to one of 24 task categories, such as opening, closing, assembly, wiping, pouring, and tool use.
  • Most task categories are covered by more than one source.
  • Annotations stop at the episode-level category. There are no labels for force, contact location, or contact timing.

• Processing and usage:

  • The sources are matched on behavior, not on hardware.
  • During pretraining, clips are sampled from the piezoresistive, magnetic, and visuotactile sources with equal sampling probabilities.
  • All supervision comes from the latent prediction objective.
  • The provided text does not specify explicit cropping or filtering rules beyond clip sampling and episode-level category assignment.

Method

The authors introduce LingBot-Touch, a foundational tactile encoder designed to learn unified, actionable representations from heterogeneous spatiotemporal tactile signals. To handle the scarcity and diversity of tactile data, the framework maps inputs from magnetic, piezoresistive, and visuotactile sensors into a canonical spherical space. This normalization aligns fingertips to a shared finger-local coordinate frame, ensuring that measurements from different sensor layouts can be processed uniformly.

As shown in the figure below, the architecture relies on a self-supervised Joint Embedding Predictive Architecture (JEPA) tailored for tactile data, termed Tactile-JEPA.

The model processes a normalized tactile clip $\mathbf{X} \in \mathbb{R}^{T \times N \times 3}$X∈RT×N×3, where $T$T is the number of frames and $N$N is the total number of valid taxels. Each taxel is assigned a five-dimensional address $\mathbf{a}_i = (t_i, f_i, \mathbf{p}_i)$ai​=(ti​,fi​,pi​), comprising the temporal window index, finger identity, and continuous 3D canonical position. This address is integrated into the model via five-axis Rotary Position Embeddings (RoPE), allowing the backbone to handle varying sensor layouts without sensor-specific positional tables.

At the input stage, modality-specific tokenizers convert the heterogeneous signals into a unified token space. The authors employ three distinct two-layer MLP tokenizers for magnetic, piezoresistive, and visuotactile sensing, each embedding two consecutive frames of a single taxel into a token. These tokens are then fed into a shared Haptic Transformer (HaT) backbone, which learns shared tactile representations at increasing levels of abstraction.

Training is driven by a latent space prediction objective. The input clip is partitioned into a visible set $\mathcal{C}$C and a masked set $\mathcal{M}$M. A context encoder processes only the visible tokens, while a target encoder processes the complete unmasked clip. The target encoder parameters are updated via an Exponential Moving Average (EMA) of the context encoder, with gradients stopped through the target branch. A modality-specific predictor takes the context encoder output, concatenated with learnable mask tokens, and predicts the target encoder representations for both masked and visible positions. The target representations are constructed by concatenating features from four depths of the target encoder. The model is optimized using a combined loss function:

$$\mathcal{L} = \frac{1}{|\mathcal{M}|D} \sum_{i \in \mathcal{M}} \| \hat{\mathbf{z}}_i - \mathrm{sg}(\bar{\mathbf{z}}_i) \|_1 + \frac{\lambda}{|\mathcal{C}|D} \sum_{i \in \mathcal{C}} \| \hat{\mathbf{z}}_i - \mathrm{sg}(\bar{\mathbf{z}}_i) \|_1$$L=∣M∣D1​i∈M∑​∥z^i​−sg(zˉi​)∥1​+∣C∣Dλ​i∈C∑​∥z^i​−sg(zˉi​)∥1​

where $D$D is the feature dimension, $\mathrm{sg}$sg denotes stop-gradient, and $\lambda = 0.5$λ=0.5. This objective encourages the model to infer missing contact information from the visible context while preserving observed contact details.

Because tactile signals are inherently sparse, applying a high masking ratio from the start of training can erase all contact evidence, leading to gradient explosions. To mitigate this, the authors implement a spatial mask-ratio warm-up strategy.

As illustrated in the figure below, the masking ratio increases linearly from 40% to 80% over the first 300 optimizer steps and remains fixed thereafter.

This warm-up spans a negligible fraction of the total 200,000 training updates but significantly stabilizes optimization during the initial untrained phase.

To further ensure the encoder generalizes across unseen sensor layouts rather than memorizing specific point sets, the authors apply layout-agnostic augmentation during pretraining. This includes exposing the model to multiple sampling densities by resampling native layouts at various spatial resolutions, and perturbing canonical taxel positions. The position perturbation adds offsets bounded by local spacing, $\mathbf{p}_i' = \mathbf{p}_i + \boldsymbol{\delta}_i$pi′​=pi​+δi​, where $\|\boldsymbol{\delta}_i\|_2 \leq 0.5 d_i$∥δi​∥2​≤0.5di​ and $d_i$di​ is the distance to the nearest valid taxel. These augmented coordinates replace the original positions in the token addresses, forcing the model to rely on the underlying interaction geometry rather than exact coordinate templates.

Experiment

The experiments evaluate a frozen tactile encoder with lightweight probes on two dexterous hands, one seen and one unseen during pretraining, across contact force, spatial localization, temporal dynamics, and tactile-to-video retrieval, and then test the same frozen tokens in real and simulated robot policies. Probing shows that the representation captures contact physics, transfers to unseen sensors, improves with more pretraining hours and model capacity, and depends on latent prediction, masking, and augmentation choices. Robot experiments show that adding frozen tactile tokens improves imitation learning, VLA, and reinforcement learning over raw tactile input, especially on contact-rich moments vision cannot resolve, and better probing performance carries over to downstream policy success.

The ablations show the default configuration is not optimal for the Sharpa Wave frozen-encoder tasks. Lowering spatial masking from 90% to 80% improves all five metrics, while whole-clip temporal masking and combined value plus position augmentation improve most metrics. For visuotactile signals at 30 Hz, 32-frame clips with two or four frames per token perform best. An 80% spatial mask is best on all five Sharpa Wave metrics, with notably better retrieval recall and center error than the 90% default. Extending temporal masking across the whole clip outperforms shorter temporal spans on four of five metrics. Combining value and position augmentation performs best on four of five metrics; position augmentation alone is slightly better for future force. For 30 Hz visuotactile data, 32-frame clips with two or four frames per token are best, while single-frame tokens and 8-frame tubelets are markedly worse.

Across the 26 real-robot tasks, augmenting policies with frozen LingBot-Touch tactile tokens consistently improves success over both the original policy and raw tactile signals, with mean gains of roughly 14 to 25 percentage points depending on the policy. Raw tactile signals provide smaller average benefits and can substantially hurt individual tasks, whereas the token representation helps even on an unfamiliar hand like XHand1. The representation is not uniformly better, but it is best in most task-policy pairs and at or above the original policy in nearly all cases. Frozen LingBot-Touch tokens improve average success for all four policies, with gains roughly double those from raw tactile signals, while raw tactile input can hurt performance on individual tasks such as mahjong sorting and egg transfer. The benefit transfers to an unfamiliar hand: on XHand1, raw tactile signals are nearly useless to all policies, but LingBot-Touch tokens computed from the same signal produce clear gains without sensor-specific training.

Integrating LingBot-Touch improves success rates and rewards across simulated contact-rich manipulation tasks, outperforming no-touch, binary contact, raw signal, and CNN baselines on both hand models. The advantage is largest in tasks with continuously changing contact, and training is more stable across random seeds. A frozen pretrained tactile representation appears to reduce exploration burden and leads to faster, more reliable convergence, with nut assembly being a slower-start exception. LingBot-Touch achieves the highest success rate and return across all evaluated tasks, clearly exceeding learned CNN, raw signal, binary contact, and no-touch baselines. Training stability improves sharply with LingBot-Touch, while alternatives can vary widely on contact-rich tasks such as plate and pen spinning. The largest gains occur where contact changes continuously, reflecting the value of a pretrained tactile representation for dynamic manipulation.

The first ablation validates frozen tactile encoder design choices, showing that lowering spatial masking, using whole-clip temporal masking, and combining value with position augmentation improve most metrics, while 32-frame clips with two or four frames per token suit visuotactile data. On real-robot benchmarks, frozen LingBot-Touch tokens consistently improve success over raw tactile signals and original policies across tasks and transfer effectively to an unfamiliar hand, though benefits are not uniform. Simulation experiments further confirm that LingBot-Touch outperforms no-touch, binary contact, raw signal, and CNN baselines, with the largest gains in continuously changing contact tasks and more stable training convergence.