Learning Haptic Active Compliance for Force-Aware Dexterous Manipulation

Naisheng Ye, Yinzhe Zhou, Junkai Zhao, Yuhang Lu
Checheng Yu, Zhenjie Yang, Pengwei Wang, and Hongyang Li

☁ Page ▤ Paper ⌘ Code

Abstract

Contact-rich dexterous manipulation requires robots to generate precise motions while actively regulating forces across evolving multi-contact interactions. Haptic observations reveal contact, but learning to act on them also requires supervision that captures the controller reference needed to sustain or adjust that contact. Nominal teleoperation commands can encode excessive loading, whereas observed configurations omit motion constrained by the object.

We introduce HACo, a Haptic Active Compliance policy that learns force-regulating actions directly from haptic feedback. Compliance-regulated teleoperation converts operator commands into controller-executable compliant actions that preserve motion intent while regulating interaction loads. HACo learns these compliant actions directly, with their command–state discrepancy providing auxiliary compliant-intent supervision. It integrates fingertip tactile and joint-torque feedback as complementary observations of local contact and load transmission through the articulated hand. A Compliance Grounding Module grounds action generation in the evolving haptic state through gated compliance attention, enabling closed-loop force regulation without explicit online contact modeling.

We evaluate HACo on a real-world benchmark spanning multi-contact friction, tangential interaction, fragile curved-surface contact, rotational torque, and deformable-object manipulation. Across 20 trials per task, HACo achieves an 83% mean progress score versus 35% for the strongest evaluated baseline. HACo variants without compliant-intent supervision and with nominal action targets achieve 73% and 59%, respectively. These results motivate learning both executable references and their contact-dependent command–state relationship.

Contributions

  • Complementary haptic perception. We couple fingertip tactile sensing with joint-torque feedback to represent both local contact and loads transmitted through the articulated hand.
  • Active compliance learning. We learn from force-regulated demonstrations by combining controller-executable compliant actions, compliant-intent supervision, and haptic-grounded action generation.
  • A real-world dexterous force benchmark. We evaluate force regulation across multi-contact friction, tangential interaction, fragile curved-surface contact, rotational torque, and deformable-object manipulation.

Method

HACo maps multi-view images, language, robot state, and haptic histories to executable compliant-action chunks. Compliant intent is predicted only as auxiliary supervision.

Architecture

Multi-view images, language, and robot state condition a flow-based action expert. For each finger, HACo fuses fingertip wrench and deformation features with torque signals from the same kinematic chain, then models interactions across digits as structured haptic memory. The Compliance Grounding Module lets action features query this memory through gated cross-attention. The model predicts executable compliant actions together with auxiliary compliant intent; only the compliant actions are sent to the robot.

HACo model architecture: structured haptic perception and gated compliance attention condition a flow-based action expert
HACo architecture with haptic-grounded action generation.

Force-regulated demonstrations

Motion retargeting provides nominal arm and hand references. Cartesian admittance and fingertip force regulation adjust them before execution, preserving task motion while limiting excessive loads. The resulting compliant references are recorded with synchronized visual, state, tactile, and torque observations.

Compliance-regulated teleoperation: retargeting, arm admittance, hand force regulation, and synchronized recording
Compliance-regulated teleoperation for force-aware demonstrations.

Compliant action learning

Observed motion omits commands blocked by contact, while nominal commands can apply excessive force. HACo instead learns the regulated compliant command. Its discrepancy from the observed hand state, Δqcit = qcmpt − qobst, supplies compliant-intent supervision during conditional flow matching and is never added to the executed action.

Nominal, observed, and compliant hand configurations under contact, illustrating compliant intent and the correction to the nominal command
Nominal, observed, and compliant hand configurations under contact.

Real-World Force Benchmark

The benchmark organizes everyday manipulation by the physical role that determines success. This section is also the primary inference-demo gallery: each task will show an autonomous rollout together with its task-specific success definition.

FRICTION
SHEAR
CURVATURE
TORQUE
DEFORMATION
Insert poker · multi-contact friction

Begin with two playing cards lying on the black platform. Use the right hand to pick up one card and transfer it to the left hand. Hold the first card upright and steady with the left hand. Then use the right hand to pick up the remaining card, align it with the card held in the left hand, and slide it into the same grip so that both cards are held together. Finish with both cards securely held in the left hand and the right hand released. Do not bend or drop either card.

Experiments

Main comparison

Five representative dexterous-manipulation approaches evaluated under a shared task protocol.

Task success rate by method20 evaluation trials per task
Insert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpaste
GR00T
GR00T + Tactile
ViTacFormer
T-Rex
HACo
Figure 2. Main comparison across five force-sensitive tasks.
MethodInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
GR00T3/200/201/207/204/2015%
GR00T + Tactile5/202/201/209/205/2022%
ViTacFormer0/201/200/202/201/204%
T-Rex4/206/202/2012/2011/2035%
HACo18/2017/2014/2019/2015/2083%

Table 1. Main comparison across the five real-world force-sensitive tasks.

Haptic Perception Ablation

Isolate the contributions of fingertip tactile and joint-torque feedback and compare factorized and coupled haptic representations.

Macro mean by perception configurationComplete five-task results
w/o Haptic Feedback
27%
w/o Tactile Feedback
45%
w/o Torque Feedback
68%
w/o Coupled Encoding
70%
HACo
83%
Figure 3. Haptic perception ablation.
Haptic perceptionInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
w/o Haptic Feedback7/202/203/208/207/2027%
w/o Tactile Feedback8/205/207/2013/2012/2045%
w/o Torque Feedback15/2016/2011/2014/2012/2068%
w/o Coupled Encoding17/2014/2012/2013/2014/2070%
HACo18/2017/2014/2019/2015/2083%

Table 2. Haptic perception ablation across sensing modalities and representation strategies.

Compliance Learning Ablation

Starting from nominal teleoperation targets, progressively introduce compliant actions generated by the compliance controller and auxiliary compliance-intent supervision.

Macro mean by compliant-action learning stageComplete five-task results
Nominal Action
59%
Compliant Action
73%
+ Compliance-Intent Supervision
83%
Figure 4. Compliant-action learning ablation.
ConfigurationCompliant ActionCISInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
Nominal Action——12/2011/209/2014/2013/2059%
Compliant Action✓—15/2015/2013/2016/2014/2073%
+ Compliance-Intent Supervision✓✓18/2017/2014/2019/2015/2083%

Table 3. Progressive composition of compliant-action learning and compliance-intent supervision.

Compliance Grounding Ablation

Hold haptic observations and active compliance actions fixed while varying how the haptic state conditions action generation.

Macro mean by compliance grounding mechanismComplete five-task results
Visuo–Haptic Fusion
41%
Action-Suffix Fusion
47%
Compliance Attention
66%
Gated Compliance Attention
83%
Figure 5. Compliance grounding ablation.
Compliance groundingInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
Visuo–Haptic Fusion10/206/207/2010/208/2041%
Action-Suffix Fusion14/209/206/2011/207/2047%
Compliance Attention14/2015/2011/2016/2010/2066%
Gated Compliance Attention18/2017/2014/2019/2015/2083%

Table 4. Compliance grounding ablation across haptic-state conditioning mechanisms.

Wrist-Camera Ablation

A compact observation study that isolates the contribution of wrist-mounted visual coverage while keeping haptic sensing and the HACo policy fixed.

Macro mean with and without wrist camerasComplete five-task results
w/o Wrist Cameras
76%
w/ Wrist Cameras
83%
Figure 6. Wrist-camera ablation.
Camera observationInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
w/o Wrist Cameras13/2015/2014/2018/2016/2076%
w/ Wrist Cameras18/2017/2014/2019/2015/2083%

Table 5. Wrist-camera observation ablation.

Failure Cases & Limitations

Representative unsuccessful rollouts reveal distinct limitations in contact-rich manipulation.

Failure F1 · Open Book

Insufficient Tangential Separation

Because the thumb fails to enter the middle of the page stack, it cannot establish enough tangential traction to separate the target section. The pages therefore move together instead of opening at the intended location.

Failure F2 · Draw on Balloon

Missed Contact Onset

The balloon slips out of the left hand during drawing. This occurs because the right hand does not recognize that the marker has already made contact and continues its motion, creating a disturbance that exceeds the left hand’s stabilizing grip.

Failure F3 · Insert Poker Cards

Excessive Pickup Force

Due to excessive normal pressure from two right-hand fingers, inter-card friction rises at pickup, causing two cards to be lifted as one stack rather than separating a single layer.

Failure F4 · Squeeze Toothpaste

Nozzle–Brush Misalignment

The toothpaste is dispensed beside the bristles instead of onto them. The reason is inaccurate nozzle–brush alignment: the policy produces the squeezing action without tightly coupling visual placement to haptic evidence of extrusion.

Citation

Citation details will be updated with the public preprint.

@misc{haco2027,
  title  = {Learning Haptic Active Compliance for
            Force-Aware Dexterous Manipulation},
  author = {Anonymous Authors},
  year   = {2027}
}