WiTac: Wrist-Informed Tactile Representations
for Contact-Rich Manipulation

Haotian He1,2,*Yiyang Chen1,3,*Congsheng Xu1,3Wenchao Yue1,5

Mingze Ye1,3Shuai Chen1,4Yi Yang1,3Yaping Li1,5

Haonan Wang1Weinan E2Wenzhao Lian3Zhijie Deng3

1 Sharpa Robotics2 PKU3 SJTU4 ShanghaiTech5 CUHK

* Equal contribution

Abstract

Similar tactile contacts can call for different corrections depending on the configuration of the hand, object and target. Rather than supplying touch and wrist vision separately, fusing them before policy injection better grounds tactile features in visual context, yielding more accurate contact representations. We introduce Wrist-Informed Tactile Representations (WiTac), a pretrained encoder that learns such representations before action generation. Pretrained on nearly 300 hours of SharpaDex data, WiTac deeply fuses tactile signals with hand motion and wrist views into a comprehensive contact representation. Conditioning downstream VLA policies (GR00T N1.7 and π0.5) on WiTac substantially improves task success over vision-only base policies across four contact-rich manipulation tasks (+28.8% and +15.0% on average), and further outperforms WiTac-NW by 18.8% and 7.5%, respectively. Offline analyses further show that incorporating wrist vision yields more accurate contact descriptors, stronger joint task–contact discrimination, greater feature stability under tactile noise, and clearer action-direction readout from similar contacts.

Wrist context resolves ambiguity in fingertip contact

Touch describes local contact. Wrist vision reveals how the hand, object, and target are arranged. Fusing them before policy injection grounds tactile features in visual context, yielding contact representations with more explicit spatial information.

WiTac is pretrained on SharpaDex and conditions a robot policy. A conceptual battery example illustrates alignment ambiguity and wrist context; four evaluation tasks and their success rates appear on the right.
Wrist context complements fingertip contact. WiTac is pretrained on SharpaDex v1.0 and conditions downstream policies. The battery-insertion illustration shows how wrist context helps resolve alignment ambiguity during contact; the right panel summarizes the evaluation tasks and GR00T N1.7 results.

Why is touch alone ambiguous?

Similar fingertip deformations may occur with different object orientations or target alignments. Local contact information does not fully specify the correction needed to complete the task.

Why fuse before policy injection?

WiTac lets tactile tokens query wrist visual features, so each finger’s contact is represented together with its surrounding spatial context before it reaches the action expert.

Learning wrist-informed contact tokens

WiTac combines tactile histories, hand kinematics, and wrist images into ten finger-level tokens. The encoder is pretrained through complementary self-supervised objectives, then frozen to condition a VLA policy.

WiTac architecture in three parts: (a) state-conditioned tactile histories query wrist features; (b) five objectives pretrain contact tokens; (c) frozen contact tokens condition the VLA action expert through gated cross-attention.

(a) Contact encoding in wrist context

An eight-frame tactile history is encoded for each of the ten fingers. Wrist-relative fingertip poses and robot state anchor these tactile features spatially. Finger tokens first interact with one another, then query patch features from both wrist cameras, producing ten finger-level contact tokens.

(b) Pretraining preserves touch and visual context

Five objectives in three groups train the shared representation on nearly 300 hours of SharpaDex data. The DINOv3 visual backbone remains frozen, while the tactile encoders and fusion blocks are optimized jointly.

  • Contact state and history: reconstruct contact descriptors, recent deformation changes, and fingertip-motion histories.
  • Conditional visual recovery: recover masked wrist-image features through the contact-token bottleneck.
  • Physical response: predict future fingertip wrench changes conditioned on an action prefix, only during pretraining.

(c) Gated injection into the action expert

The pretrained WiTac encoder is frozen for policy learning. Selected action blocks attend to the contact tokens through a learned gate. The same interface supports GR00T N1.7 and π0.5, preserving each policy’s native visual, language, and state inputs. Pretraining prediction heads are removed for inference.

Contact representations improve real-world manipulation

We evaluate battery insertion, bowl wiping, USB reinsertion, and cabinet opening, with 20 trials per method per task. WiTac improves average success from 37.5% to 66.3% with GR00T N1.7, and from 43.8% to 58.8% with π0.5.

Closed-loop task success rates (%)

Higher is better. Average is the unweighted mean across four tasks.

Base policyWiTac-NWWiTac

WiTac-NW removes wrist fusion from the contact encoder. Its downstream VLA still receives its native visual inputs. The displayed averages and gains follow the manuscript’s rounding.

Full comparison with all reported baselines
Closed-loop success rates (%)
PolicyInsert batteryWipe bowlReinsert USBOpen cabinetAverage
FTP-1405556040.0
T-Rex4565157550.0
GR00T N1.73040107037.5
+ WiTac-NW456058047.5
+ VT-MUSE joint representation4070157550.0
+ WiTac5580409066.3
π0.55565104543.8
+ WiTac-NW6065156551.3
+ WiTac7075207058.8

WiTac and WiTac-NW in action

Compare GR00T N1.7 rollouts across four tasks: successful examples use WiTac, and unsuccessful examples use WiTac-NW. Select a task to compare the recordings.

How does wrist context improve tactile representations?

More accurate contact reconstruction

Across 9,632 validation windows from 602 episodes, WiTac reconstructs contact area, intensity, and center more accurately than WiTac-NW. Wrist context helps preserve the geometry and strength of local touch.

Relative reconstruction error reduction versus WiTac-NW: contact area 7.57%, intensity 10.00%, and center 2.64%. Higher is better.
Relative reconstruction error reduction versus WiTac-NW. Area and intensity use the all-window evaluation. Higher is better.

Joint readout of task context and contact state

Frozen WiTac features distinguish both the task and its contact state. Across 12- and 20-class evaluations on two test splits, WiTac improves macro-F1 over wrist-only DINO and WiTac-NW. It is comparable to simple feature concatenation in three settings, with a larger gain on the 20-class confirmation split.

12-class evaluation

12-class joint task-contact macro-F1. Original split: DINO 71.40%, WiTac-NW 71.66%, concatenation 78.28%, WiTac 79.31%. Confirmation split: DINO 73.22%, WiTac-NW 71.89%, concatenation 77.87%, WiTac 77.61%.
Macro-F1 (%) on the original (3,487 windows) and confirmation (3,233 windows) splits. Higher is better. Concat. combines WiTac-NW and DINO features.

20-class evaluation

20-class joint task-contact macro-F1. Original split: DINO 48.94%, WiTac-NW 56.00%, concatenation 57.92%, WiTac 57.83%. Confirmation split: DINO 49.35%, WiTac-NW 55.78%, concatenation 54.63%, WiTac 59.49%.
Macro-F1 (%) on the original (3,555 windows) and confirmation (3,288 windows) splits. Both class-count pages use the same 0–100% scale.

More stable features under tactile noise

With Gaussian noise added to tactile inputs (σ = 3 on the 8-bit intensity scale), WiTac reduces feature drift across all four tasks. GR00T N1.7 action error also falls by 7.35–22.76%; action-level gains for π0.5 vary by task.

Relative reductions versus WiTac-NW under tactile noise. Battery: feature drift +37.4%, GR00T action MAE +22.76%, pi 0.5 action MAE −0.56%. Bowl: +35.6%, +7.35%, −1.90%. USB: +37.0%, +9.18%, +1.91%. Cabinet: +44.1%, +21.59%, +2.19%.
Relative reductions (%) versus WiTac-NW under Gaussian tactile noise (σ = 3). Positive values indicate improvement; negative values indicate higher action error. Wrist images and robot state are unchanged.

Clearer action-direction readout from similar contacts

Similar tactile contacts can precede different corrective motions. On 25 contact-matched test pairs per task, WiTac achieves the strongest mean directional alignment across all four tasks using equal-capacity linear probes on frozen features.

Mean action-direction cosine similarity for DINO, WiTac-NW, and WiTac, respectively. Battery: 0.6814, 0.6794, 0.7325. USB: 0.3281, 0.3621, 0.4295. Bowl: 0.7246, 0.7696, 0.7881. Cabinet: 0.6993, 0.6669, 0.7386.
Mean cosine similarity on 25 contact-matched test pairs per task, using equal-capacity linear probes on frozen features. Higher is better. These are offline direction-readout scores, not closed-loop success rates.

Reconstruction videos across four tasks

Watch synchronized camera views, wrist-feature reconstruction, and ten-finger tactile history. Each clip pairs ground truth (GT) with WiTac. Tactile rows show GT and WiTac for the historical change T(t) − T(t−8), using the same fixed ±40 color scale. Wrist panels show masked DINO feature reconstruction.

Battery insertion4 s · 1× speed · 1080p

Battery transport and insertion onset with synchronized ten-finger tactile reconstruction.

USB reinsertion4 s · 1× speed · 1080p

Connector extraction and return with synchronized tactile reconstruction.

Bowl wiping6 s · 1× speed · 1080p

Continuous sponge-on-bowl wiping with reconstruction across both hands.

Cabinet opening3.2 s · 1× speed · 1080p

The cabinet door rotates open while fingertip contact is reconstructed.

Citation

If you find our work useful, please consider citing our arXiv paper:

@article{he2026witac,
  title={{WiTac}: Wrist-Informed Tactile Representations for Contact-Rich Manipulation},
  author={He, Haotian and Chen, Yiyang and Xu, Congsheng and Yue, Wenchao and
          Ye, Mingze and Chen, Shuai and Yang, Yi and Li, Yaping and
          Wang, Haonan and E, Weinan and Lian, Wenzhao and Deng, Zhijie},
  journal={arXiv preprint arXiv:2611.21057},
  year={2026}
}