RoboAssist

Interactive Human–Humanoid Planning
for Long-Horizon Surgical Assistance

Abstract

Long-horizon surgical assistance requires humanoid robots to coordinate with evolving human activities while maintaining safety across planning and execution. We present RoboAssist, an agent-based framework for interactive human–humanoid planning that integrates workflow reasoning, task coordination, and cross-layer safety. At its core is an asymmetric dual-track representation that separates partially observed human process states from executable robot task sequences. By jointly updating human-process estimates, scene context, and task dependencies online, RoboAssist adapts the remaining robot plan as workflow requirements and environmental conditions evolve.

A cross-layer safety architecture combines preventive navigation regulation, reactive regulation during close-range handover, and independent whole-body runtime supervision. We demonstrate the framework on a Unitree G1 humanoid robot in long-horizon, multi-stage simulated surgical assistance scenarios encompassing multimodal interaction, instrument handling, medical material transport, navigation, and safe human–robot handover. Experiments show improved multi-stage task completion and dynamic adaptation over task-matched baselines, with the clearest gains under workflow-request changes.

RoboAssist Framework

RoboAssist framework integrating interaction context, interactive planning, skill execution, and runtime safety constraints.

Overview of RoboAssist: RoboAssist is an agent-based framework for long-horizon human–humanoid surgical assistance. Human instructions and scene observations update workflow knowledge and task memory, while an LLM-driven agent aligns partially observed human-process states with executable robot tasks through evidence gates. When requirements change, the agent preserves the valid task prefix and replans only the affected suffix, then dispatches navigation, grasping, placement, and handover skills under continuous runtime safety constraints.

Video

Experiment Result

Task success rates and sequential subtask completion for RoboAssist and comparison methods.
Task execution performance. RoboAssist achieved 93.3% single-item success and 85.7% long-horizon task success.
Workflow completion-time distributions for four planning methods.
Workflow completion time. Completion-time distributions over five successful multi-object runs per method.

Request-order adaptation vs. FullReplan

81.25% success · 16 trials
−66.9% update latency
−43.0% token use

Runtime Safety

RoboAssist cross-layer safety architecture covering surgeon-aware navigation, VLA handover regulation, and whole-body runtime supervision.
Cross-layer safety architecture. Preventive navigation, reactive handover regulation, and fail-safe whole-body supervision.
Cross-layer safety evaluation showing the navigation profile, protective response rate, and pathway response times.
Cross-layer safety evaluation. Runtime supervision improved timely visual-blind-zone responses from 2/5 to 5/5; mean response times were 20.11 ms for force feedback and 4.833 s for the VLM pathway.

BibTeX

@misc{roboassistTODO,
  title  = {RoboAssist: Interactive Human--Humanoid Planning for
            Long-Horizon Surgical Assistance},
  author = {TODO},
  year   = {TODO},
  url    = {TODO}
}