RoboAssist

Interactive Human–Humanoid Planning
for Long-Horizon Surgical Assistance

Abstract

Long-horizon surgical assistance requires humanoid robots to coordinate with evolving human activities while maintaining safety across planning and execution. We present RoboAssist, an agent-based framework for interactive human–humanoid planning that integrates workflow reasoning, task coordination, and cross-layer safety. At its core is an asymmetric dual-track representation that separates partially observed human process states from executable robot task sequences. By jointly updating human-process estimates, scene context, and task dependencies online, RoboAssist adapts the remaining robot plan as workflow requirements and environmental conditions evolve.

A cross-layer safety architecture combines preventive navigation regulation, reactive regulation during close-range handover, and independent whole-body runtime supervision. We demonstrate the framework on a Unitree G1 humanoid robot in long-horizon, multi-stage simulated surgical assistance scenarios encompassing multimodal interaction, instrument handling, medical material transport, navigation, and safe human–robot handover. Experiments show improved multi-stage task completion and dynamic adaptation over task-matched baselines, with the clearest gains under workflow-request changes.

Keywords: Robot Agent, Interactive Planning, Humanoid Assistance

RoboAssist Framework

RoboAssist framework integrating interaction context, interactive planning, skill execution, and runtime safety constraints.

Overview of RoboAssist: RoboAssist is an agent-based framework for long-horizon human–humanoid surgical assistance. Human instructions and scene observations update workflow knowledge and task memory, while an LLM-driven agent aligns partially observed human-process states with executable robot tasks through evidence gates. When requirements change, the agent preserves the valid task prefix and replans only the affected suffix, then dispatches navigation, grasping, placement, and handover skills under continuous runtime safety constraints.

Robot Demonstrations

Explore long-horizon task planning, interactive replanning, and safe human–robot assistance through synchronized experimental views.

Give RoboAssist a Surgical Assistance Task

This illustrative demonstration shows how a human request unfolds into a sequence of coordinated surgical-assistance tasks. Using the delivery of a medical item as an example, it visualizes navigation to the supply table, grasping, transport, and handover to the surgeon, allowing the human workflow to continue. The sequence illustrates the connection between task-level planning and robot skill execution; recorded experiments are presented in the demonstration videos above.

Unitree G1 · Surgical assistance Interactive 3D scene
Loading surgical workspace…
Ready for instruction
Drag to orbit
RoboAssist Loading
  1. 1Receive the request
  2. 2Navigate to the sterile table
  3. 3Pick up gauze
  4. 4Carry gauze to the surgeon
  5. 5Hand over to the surgeon
  6. 6Place gauze on the operating table
  7. 7Resume the surgical task

SPACE & CONTEXT

Semantic Mapping for Planning and Navigation

From handheld RGB-D scans to persistent scene memory for online agent planning and robot semantic navigation.

1Offline-to-online workflow

OFFLINE Build persistent scene memory

Handheld RGB-D Scan

Real surgical workspace recorded with a handheld RGB-D camera.
RGBDepthCamera poses

Capture the workspace from multiple viewpoints.

Object-level 3D Mapping

Reconstructed point cloud of the scanned surgical workspace.

Object segmentation 3D projection Multi-view association & fusion

Offline Semantic Memory

Existing surgical-workspace semantic map with category-colored objects and no browser controls.
Object identity3D locationSpatial context

Store an object-level map for later retrieval.

Memory retrieval
ONLINE Ground instructions and execute tasks

Agent Online Planning

Human instructionPlease bring me gauze.

+ Current observation

Retrieve scene memory to identify the target and generate or adjust the task plan.

  1. Retrieve gauze
  2. Navigate
  3. Grasp
  4. Deliver

Semantic Goal Grounding

  1. Retrieve the object's map location
  2. Determine a reachable approach pose
  3. Pass the goal to the navigation module

Robot Semantic Navigation

  1. 01Navigation goal
  2. 02Path planning
  3. 03Robot execution

Navigate to the approach pose, then continue with grasping and delivery skills.

Execution state Agent
Execution feedback
Built offline, the semantic map provides persistent scene memory. During execution, the agent retrieves object and spatial context to ground instructions and plan tasks; navigation uses a reachable approach pose, and execution feedback informs subsequent planning.

2Interactive 3D preview

Explore full screen
Reconstructed surgical workspace with three functional regions

Explore the workspace in 3D. Select a region or adjust the slider to change the view.

Explore the semantic map

Semantic Map

We build an offline semantic map from a handheld RGB-D scan by segmenting objects in RGB images, lifting them into 3D using depth and camera poses, and fusing observations across views. The map stores object semantics, locations, and spatial relationships as persistent scene memory for the planning agent. By retrieving this memory, the agent grounds language instructions in the workspace and identifies relevant objects and destinations for navigation, manipulation, and long-horizon action planning.

Room reconstructed point cloud in the RoboAssist semantic scene explorer
ROOM · OBJECT-LEVEL RECONSTRUCTION

A scene you can explore.

29 objects · 15 categories · 284,293 semantic points

Rotate the room. Select an object. Inspect its geometry.

Drag to orbit, scroll to zoom, and select an object to inspect it. Switch between original RGB and semantic colors.

Open full explorer ↗

Full Video

This video presents RoboAssist’s basic human–robot interaction, interactive task planning, and long-horizon surgical assistance in a simulated surgical workspace. It shows how human requests and request updates guide the agent’s coordination of navigation, medical-item handling, and handover. Dialogue, task-state visualizations, and multiple camera views illustrate the connection between online planning and robot execution.

Experiment Result

Task success rates and sequential subtask completion for RoboAssist and comparison methods.
Task execution performance. RoboAssist achieved 93.3% single-item success and 85.7% long-horizon task success.
Workflow completion-time distributions for four planning methods.
Workflow completion time. Completion-time distributions over five successful multi-object runs per method.

Request-order adaptation vs. FullReplan

81.25% success · 16 trials
−66.9% update latency
−43.0% token use

Cross-Layer Safety

Level 1 · Prevent

Preventive surgeon-aware navigation regulation

  1. Far Nominal speed
  2. Near Boundary Slowed speed
  3. Stop Boundary / contact

Protected operating region Ωop

View original architecture figure
RoboAssist cross-layer safety architecture covering surgeon-aware navigation, VLA handover regulation, and whole-body runtime supervision.
Cross-layer safety architecture. Preventive navigation, reactive handover regulation, and fail-safe whole-body supervision.
Cross-layer safety evaluation showing the navigation profile, protective response rate, and pathway response times.
Cross-layer safety evaluation. Runtime supervision improved timely visual-blind-zone responses from 2/5 to 5/5; mean response times were 20.11 ms for force feedback and 4.833 s for the VLM pathway.

Scope and Future Work

Our Scope

Long-horizon human–humanoid assistance

Coordinates multi-stage assistance as requests and scenes evolve.

Task-level planning

Plans tasks while robot motion remains skill-based.

Interactive alignment and replanning

Aligns human-process states with robot tasks through evidence gates.

Cross-layer safety

Covers navigation, handover, and whole-body execution.

Future Work

Future work will extend the evaluation to longer and more diverse workflows, broaden interaction and environmental variations, and improve VLA skill robustness through broader training coverage. We also aim to enable human–humanoid collaboration in real surgical settings and progress toward fully autonomous surgical workflows.

FAQ

RoboAssist targets long-horizon surgical assistance in which a humanoid robot must coordinate navigation, manipulation, material transport, and handover while human requests and workflow states may change during execution.

RoboAssist explicitly separates partially observed human-process states from executable robot tasks. These two tracks are connected through evidence-gated dependencies, allowing the robot to reason about human progress without treating predicted human actions as directly executable robot states.

RoboAssist identifies the earliest affected unfinished task, preserves the still-valid prefix of the existing plan, and regenerates only the affected suffix. This residual replanning avoids unnecessarily rebuilding the entire remaining task sequence.

No. Predicted human-process states may trigger task generation or preparatory actions, but predictions alone cannot satisfy an evidence gate. A human-process condition is confirmed only when its corresponding evidence criterion is satisfied.

No. RoboAssist performs task-level planning. Low-level robot behavior is encapsulated in reusable parameterized skills such as navigation, grasping, placement, and handover, which are dispatched through the skill executor.

RoboAssist uses a three-layer safety architecture. Preventive navigation regulation slows or stops the robot near the surgeon's protected operating region; reactive interaction regulation can interrupt unsafe handover motion; and an independent runtime supervisor can trigger a latched emergency stop when abnormal force or joint-motion states are detected.

Not yet. The current study evaluates RoboAssist in a simulated surgical-assistance setting on a Unitree G1 humanoid. Clinical deployment, broader workflow diversity, and validation under more varied interaction and environmental conditions remain outside the scope of the present evaluation.

BibTeX

@misc{roboassistTODO,
  title  = {RoboAssist: Interactive Human--Humanoid Planning for
            Long-Horizon Surgical Assistance},
  author = {TODO},
  year   = {TODO},
  url    = {TODO}
}