HACo: Learning Haptic Active Compliance for Force-Aware Dexterous Manipulation

Naisheng Ye1,2,†, Yinzhe Zhou2,3, Junkai Zhao2, Yuhang Lu1,2
Checheng Yu1, Zhenjie Yang1, Pengwei Wang2, and Hongyang Li1

1The University of Hong Kong
2Beijing Academy of Artificial Intelligence (BAAI)
3Johns Hopkins University

†Work done during an internship at BAAI.

☁ Page ▤ Paper Code · Coming soon

Abstract

Contact-rich dexterous manipulation requires policies to translate physical feedback into motion commands that regulate interaction loads across evolving multi-contact interactions. This requires both haptic observations that capture the contact state and action supervision that demonstrates how motion commands should adapt to it. Existing policies often overlook the complementary roles of fingertip tactile sensing and joint torque. Meanwhile, commonly used action targets include nominal teleoperation commands, which can encode excessive loading, and observed configurations, which omit motion constrained by the object.

We introduce HACo, a Haptic Active Compliance policy that learns force-regulating actions directly from haptic feedback. Compliance-regulated teleoperation converts operator commands into controller-executable compliant actions that preserve motion intent while regulating interaction loads. HACo learns these compliant actions directly, with their command-state discrepancy providing auxiliary compliant-intent supervision. HACo further combines local contact responses from fingertip tactile sensing with load transmission through the articulated hand captured by joint-torque feedback, including contacts beyond tactile coverage. A Compliance Grounding Module grounds action generation in the evolving haptic state through gated haptic cross-attention, enabling closed-loop force regulation without explicit online contact modeling.

We evaluate HACo on a real-world benchmark spanning multi-contact friction, tangential interaction, fragile curved-surface contact, rotational torque, and deformable-object manipulation. Across 20 trials per task, HACo achieves an 83% mean success rate versus 35% for the strongest evaluated baseline. These results demonstrate HACo's ability to translate complementary haptic feedback into active compliance across diverse force-sensitive dexterous manipulation tasks.

Contributions

  • We develop complementary haptic perception that couples fingertip tactile sensing with joint-torque feedback to represent both local contact and loads transmitted through the articulated hand.
  • We formulate active compliance learning from regulated demonstrations, combining controller-executable compliant actions, compliant-intent supervision, and the Compliance Grounding Module for haptic-conditioned action generation.
  • We introduce a real-world dexterous force benchmark spanning multi-contact friction, tangential interaction, fragile curved-surface contact, rotational torque, and deformable-object manipulation.

Method

HACo learns controller-executable compliant actions from force-regulated demonstrations and conditions action generation on haptic feedback.

Architecture

HACo conditions a flow-based action expert on multi-view images, language, robot state, and haptic histories. Its haptic expert first aligns fingertip wrench and deformation features with joint-torque features from the same finger, then fuses information across fingers into a structured haptic representation. The Compliance Grounding Module lets action features query this representation through gated haptic cross-attention alongside vision-language conditioning. Trained with conditional flow matching, the action expert predicts controller-executable compliant actions with auxiliary compliant-intent supervision.

Overview of HACo with kinematics-aligned haptic perception and gated compliance grounding
Figure 1. Overview of HACo with kinematics-aligned haptic perception and gated compliance grounding.

Force-regulated demonstrations

Motion retargeting provides nominal arm and hand references. Cartesian admittance and fingertip force regulation adjust them before execution, preserving task motion while limiting excessive loads. The resulting compliant references are recorded with synchronized visual, state, tactile, and torque observations.

Compliance-regulated teleoperation: retargeting, arm admittance, hand force regulation, and synchronized recording
Figure 2. Compliance-regulated teleoperation for force-aware demonstrations.

Compliant action learning

Observed motion omits commands blocked by contact, while nominal commands can apply excessive force. HACo instead learns the regulated compliant command. Its discrepancy from the observed hand state, Δqcit = qcmpt − qobst, supplies compliant-intent supervision during conditional flow matching.

Nominal, observed, and compliant hand configurations under contact, illustrating compliant intent and the correction to the nominal command
Figure 3. Hand action semantics under contact: nominal, observed, and compliant configurations.

Real-World Force Benchmark

Most prior work evaluates dexterous manipulation primarily through motion outcomes rather than force regulation. Our benchmark therefore comprises five multi-stage bimanual tasks spanning multi-contact friction, precise tangential-force control, fragile curved-surface contact, rotational torque, and deformable-object compression. While each task emphasizes a distinct interaction regime, all require the policy to complete the intended motion while regulating the physical interaction that makes completion possible.

FRICTION
SHEAR
CURVATURE
TORQUE
DEFORMATION
Insert poker · multi-contact friction

Begin with two playing cards lying on the black platform. Use the right hand to pick up one card and transfer it to the left hand. Hold the first card upright and steady with the left hand. Then use the right hand to pick up the remaining card, align it with the card held in the left hand, and slide it into the same grip so that both cards are held together. Finish with both cards securely held in the left hand and the right hand released. Do not bend or drop either card.

Experiments

We evaluate whether HACo improves force-sensitive dexterous manipulation over visuomotor and tactile-augmented policies, then isolate the contributions of haptic perception, compliant-action learning, compliance grounding, and wrist-mounted visual observations. Every entry reports successful physical rollouts out of 20; Mean is the macro-average across the five tasks.

Main comparison

Compare HACo with visuomotor and tactile-augmented policies under the same five-task, 20-rollout evaluation protocol.

Task success rate by method20 evaluation trials per task
Insert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpaste
GR00T
GR00T + Tactile
ViTacFormer
T-Rex
HACo
Table 1a. Task-wise success-rate summary across the five force-sensitive tasks.
MethodInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
GR00T3/200/201/207/204/2015%
GR00T + Tactile5/202/201/209/205/2022%
ViTacFormer0/201/200/202/201/204%
T-Rex4/206/202/2012/2011/2035%
HACo18/2017/2014/2019/2015/2083%

Table 1b. Numerical policy comparison on the real-world force-sensitive dexterous manipulation benchmark.

Insight

HACo achieves an 83% mean success rate and leads on all five tasks, exceeding T-Rex by 48 percentage points and GR00T by 68. It also reduces mean fingertip force by 19% relative to GR00T, showing that the higher task success is accompanied by more compliant interaction. The modest gain from GR00T to GR00T + Tactile (15% to 22%) further indicates that simple tactile concatenation is not enough; this favors connecting haptic feedback to the evolving action representation.

Haptic perception ablation

Isolate the contributions of fingertip tactile and joint-torque feedback and compare factorized and coupled haptic representations.

Macro mean by perception configurationComplete five-task results
w/o Haptic Feedback
27%
w/o Tactile Feedback
45%
w/o Torque Feedback
68%
w/o Coupled Encoding
70%
HACo
83%
Table 2a. Mean success-rate summary by haptic-perception configuration.
Haptic perceptionInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
w/o Haptic Feedback7/202/203/208/207/2027%
w/o Tactile Feedback8/205/207/2013/2012/2045%
w/o Torque Feedback15/2016/2011/2014/2012/2068%
w/o Coupled Encoding17/2014/2012/2013/2014/2070%
HACo18/2017/2014/2019/2015/2083%

Table 2b. Numerical results for the haptic-perception ablation.

Insight

Removing all haptic feedback drops the mean success rate from 83% to 27%, confirming that physical observations are essential. Tactile feedback alone reaches 68%, compared with 45% for joint torque alone, so local contact is the dominant cue. Joint torque remains complementary: adding it improves every task and raises cap removal from 70% to 95%. Coupled encoding provides a further gain from 70% to 83%, supporting kinematics-aligned fusion across the hand.

Compliance learning ablation

Progressively introduce controller-executable compliant actions and auxiliary compliant-intent supervision from regulated demonstrations.

Macro mean by compliant-action learning stageComplete five-task results
Nominal Action
59%
Compliant Action
73%
+ Compliance-Intent Supervision
83%
Table 3a. Mean success-rate summary by compliant-action learning stage.
ConfigurationCompliant ActionCISInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
Nominal Action——12/2011/209/2014/2013/2059%
Compliant Action✓—15/2015/2013/2016/2014/2073%
+ Compliance-Intent Supervision✓✓18/2017/2014/2019/2015/2083%

Table 3b. Numerical results for the compliance-learning ablation.

Insight

Keeping the compliant action target but removing compliant-intent supervision lowers the mean from 83% to 73%. Replacing that target with the nominal teleoperation command lowers it further to 59%. The larger loss from changing the action target shows that the controller-executable, force-regulated reference is the primary contribution, while compliant-intent supervision adds a complementary benefit by preserving the force-regulating command-state discrepancy.

Compliance grounding ablation

Hold haptic observations and compliant action targets fixed while varying how haptic state conditions action generation.

Macro mean by compliance grounding mechanismComplete five-task results
Visuo–Haptic Fusion
41%
Action-Suffix Fusion
47%
Compliance Attention
66%
Gated Compliance Attention
83%
Table 4a. Mean success-rate summary by compliance-grounding mechanism.
Compliance groundingInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
Visuo–Haptic Fusion10/206/207/2010/208/2041%
Action-Suffix Fusion14/209/206/2011/207/2047%
Compliance Attention14/2015/2011/2016/2010/2066%
Gated Compliance Attention18/2017/2014/2019/2015/2083%

Table 4b. Numerical results for the compliance-grounding ablation.

Insight

Direct Visuo-Haptic Fusion and Action-Suffix Fusion reach only 41% and 47%, while dedicated haptic cross-attention reaches 66% and the complete gated CGM reaches 83%. Cross-attention gives the evolving action representation explicit access to contact evidence without disrupting the pretrained visual or action streams. The zero-initialized gate adds another 17 points by preserving pretrained computation at the start of adaptation and regulating reliance on noisy or intermittent haptic signals.

Wrist-camera ablation

Remove both wrist cameras while keeping the haptic observations and HACo policy otherwise unchanged.

Macro mean with and without wrist camerasComplete five-task results
w/o Wrist Cameras
76%
w/ Wrist Cameras
83%
Table 5a. Mean success-rate summary with and without wrist cameras.
Camera observationInsert pokerOpen bookDraw on balloonUnscrew capSqueeze toothpasteMean
w/o Wrist Cameras13/2015/2014/2018/2016/2076%
w/ Wrist Cameras18/2017/2014/2019/2015/2083%

Table 5b. Numerical results for the wrist-camera ablation.

Insight

Removing both wrist cameras reduces the mean success rate by only 7 points, from 83% to 76%. This modest drop suggests that haptic feedback recovers contact state that is difficult to observe under visual occlusion. Wrist views remain useful, but the result indicates that sensing anchored at the interaction interface provides a more direct and robust contact representation.

Failure Cases & Limitations

Representative unsuccessful rollouts reveal distinct limitations in contact-rich manipulation.

Failure F1 · Open Book

Insufficient Tangential Separation

Because the thumb fails to enter the middle of the page stack, it cannot establish enough tangential traction to separate the target section. The pages therefore move together instead of opening at the intended location.

Failure F2 · Draw on Balloon

Missed Contact Onset

The balloon slips out of the left hand during drawing. This occurs because the right hand does not recognize that the marker has already made contact and continues its motion, creating a disturbance that exceeds the left hand’s stabilizing grip.

Failure F3 · Insert Poker Cards

Excessive Pickup Force

Due to excessive normal pressure from two right-hand fingers, inter-card friction rises at pickup, causing two cards to be lifted as one stack rather than separating a single layer.

Failure F4 · Squeeze Toothpaste

Nozzle–Brush Misalignment

The toothpaste is dispensed beside the bristles instead of onto them. The reason is inaccurate nozzle–brush alignment: the policy produces the squeezing action without tightly coupling visual placement to haptic evidence of extrusion.

Citation

If you find our work helpful, please cite it below.

@misc{ye2026hacolearninghapticactive,
  title         = {HACo: Learning Haptic Active Compliance for Force-Aware Dexterous Manipulation},
  author        = {Naisheng Ye and Yinzhe Zhou and Junkai Zhao and Yuhang Lu and Checheng Yu and Zhenjie Yang and Pengwei Wang and Hongyang Li},
  year          = {2026},
  eprint        = {2609.36596},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2609.36596}
}

This website draws inspiration from the open-source ENPIRE project-page template and the T-Rex project page. We thank both teams for sharing their work.