dsh-ros2

ROS2 debugging toolset and robot-state vision analysis for DeepSeek Harness (DSH), shipped as a plugin. 中文版见 README_CN.md.

dsh-ros2 gives a DSH agent full robot development / debugging capabilities on any host with ROS2, organized in four capability tiers:

All tools run plain ros2 / colcon / rosdep CLI commands on the host; L1 never modifies anything, L2 always asks first.

Screenshots

RViz2 offscreen render (latest lite_urdf, real material colors)	Head camera	Wrist-left camera	Wrist-right camera

Left: rviz_offscreen_node renders with the real rviz stack (OGRE) and publishes to the /rviz/scene image topic. Right: three frames grabbed from live camera topics by ros2_image_snapshot (1280×720). Full test record: docs/test-robot-state-vision.md.

Features

• Zero-intrusion diagnostics: 51 tools cover most ROS2 debugging scenarios — from "is the package installed?" to "what is on this topic right now?", one command, one answer;
• Whole-graph topology: ros2_graph folds nodes/publishers/subscribers/services/actions into one JSON — see the system structure in seconds;
• Approval-gated writes: builds, dependency installs, message scaffolding etc. go through the DSH approval service; fail-closed, denial = failure;
• Visualization as a service: "see" headlessly — screenshots / multimodal description / window interaction are fully local, no remote display;
• Parallel realtime vision: the VLM runs in a separate ROS2 process (vlm_node, service /vlm/describe); images come from topics (sensor_msgs/Image / CompressedImage); vision_bringup auto-creates one bridge per image topic, headless-ready;
• RViz2 offscreen rendering (motion render ~22 Hz on llvmpipe; 30 Hz full rate with GPU): the real rviz render kernel (rviz_common + OGRE) renders any .rviz scene on a virtual display and publishes it as an image topic — no screenshots, no X11 window-stacking dependency. Performance optimizations: open3d low-poly meshes (scripts/simplify_visual_meshes.py) + direct OGRE pixel read (no PNG round-trip) + double-render elimination → motion rendering 1.9 → ~22 Hz on llvmpipe (11×), and 30 Hz full rate with NVIDIA GPU passthrough (v0.9.3), memory −2.5×;
• Real-time safety framework (safety_monitor node + robot_safety_* tools, see Safety framework): layered defense — tool-layer safety gate (pre-execution /safety/state check) → reactive monitors (motion tracking/stall with hysteresis, joint-feedback loss, watchdog, optional torque) → event-driven VLM semantic arbitration → human arbitration. Latched NORMAL/LOCKED state machine (lock persists until a human unlocks; non-fatal events never lock); every threshold/topic/lock-action is registered per robot in the profile safety section; the geometric pre-check (joint limits / velocity / FK self-collision) is a reserved layer;
• Bundled skills (4): ros2-diagnostics (which tool to use when, narrow-down methodology), robot-state-vision-analysis (status → offscreen render → VLM → cross-check), robot-registration (first-contact body profile + topology baseline) and robot-retrieval (instant profile load and bring-up).

Quick start

Requirements

• A host with ROS2 (Jazzy verified; Humble should work), ros2 on PATH;
• Node ^22.19 || >=24 (DSH host requirement);
• L4 vision additionally needs Python 3 rclpy and an OpenAI-compatible VLM gateway (e.g. Gemini or a self-hosted gateway).

Install the plugin

dsh plugin --profile <profile> add dsh-ros2

Minimal configuration

# fragment of a DSH profile plugin config
- insert:
    - id: dsh-ros2
      name: dsh-ros2
      config:
        rosSetup: source /opt/ros/jazzy/setup.bash &&   # prepare the ROS2 environment
        workspaceRoot: /home/you/ros2_ws                 # default cwd for colcon/rosdep
        vision:
          provider: gemini                               # mock | gemini | openai
          apiKey: ${GEMINI_API_KEY}                      # inject via env/secret manager, never hard-code
        safetyStrict: warn                               # motion-tool gate when the safety monitor is unreachable: 'warn' (default) | 'reject' (fail-closed); LOCKED always rejects

Three-minute taste

ros2_graph                          # whole system topology in one shot
ros2_topic_list                     # all topics and types
ros2_topic_echo /joint_states       # sample one frame of joint states
ros2_tf_list                        # TF tree edges
ros2_doctor                         # system health report

Tool reference

L1 read-only diagnostics

L2 management (approval-gated)

Every L2 tool performs a write operation and asks the user first via the DSH approval service (fail-closed when unavailable or denied). The read-only helpers ros2_jobs_list / ros2_job_status need no approval.

L3 visualization

GUI lifecycle + screenshots + multimodal vision ("see first, then move") + xdotool-level interaction ("see and move"). Screenshots use Pillow ImageGrab on X11 (no extra CLI install); vision providers are pluggable; interaction requires xdotool (sudo apt install xdotool).

Interaction recipes (model-facing): orbit the RViz2 view with ros2_gui_interact {action: "drag", windowTitle: "rviz2", button: 1, toX: <dx>, toY: <dy>}, zoom with action: "drag", button: 3, reload a display config with ros2_gui_interact {action: "key", keys: "ctrl+shift+r"}. When wmctrl cannot enumerate windows (e.g. no window manager on the display), window-relative interaction reports "window not found" — fall back to absolute screen coordinates. Interaction is local to the host session (no approval, same as other L3 tools).

L4 realtime vision (parallel VLM over ROS2, headless image topics)

Perception matches the robot-control stack: the VLM runs in a separate ROS2 process (vlm_node, service /vlm/describe + cached topic /vlm/description), and images come from sensor_msgs/Image topics, never X11 screenshots — headless-ready. Requires the dsh_ros2_vlm ROS2 package (vlm/); build/run see docs/architecture.md §4.

# build + launch the vision pipeline (auto bridge per image topic)
mkdir -p /tmp/vlm_ws/src && ln -s <repo>/vlm /tmp/vlm_ws/src/dsh_ros2_vlm
cd /tmp/vlm_ws && colcon build --symlink-install && source install/setup.bash
VLM_API_KEY=... ros2 run dsh_ros2_vlm vlm_node &       # parallel VLM process
ros2 run dsh_ros2_vlm vision_bringup &                 # discover topics, one bridge each

RViz2 offscreen rendering (dsh_ros2_rviz_offscreen)

The real rviz rendering stack (rviz_common + OGRE + rviz_default_plugins) loads a .rviz scene offscreen under Xvfb and publishes it to the /rviz/scene image topic — read from the render kernel, not X screenshots, no window-stacking dependency.

# build (needs a colcon workspace like vlm_ws)
ln -s <repo>/offscreen /tmp/vlm_ws/src/dsh_ros2_rviz_offscreen
cd /tmp/vlm_ws && colcon build --symlink-install && source install/setup.bash
# run (config_path points at a .rviz scene file)
xvfb-run -a -s "-screen 0 1280x800x24" ros2 run dsh_ros2_rviz_offscreen rviz_offscreen_node \
  --ros-args -p config_path:=/tmp/robot_scene.rviz -p topic:=/rviz/scene \
  -p width:=800 -p height:=600 -p rate:=5.0

Robot body mesh rendering essentials (pitfalls collected; details in docs/architecture.md §4.4):

1. Jazzy RobotModel properties: use Description Source: Topic + Description Topic: <topic> (the legacy Robot Description: is ignored → Links empty);
2. mesh paths: use absolute paths or a file:// prefix in the URDF (bare paths fail in resource_retriever); the description publisher must stay alive (transient-local, late subscribers miss it otherwise);
3. URDF must bind to TF by name: URDF link names must exactly match the live TF frame names (publishing the robot's actual /robot_description works). Mismatch → every link transform lookup fails → all meshes pile at the fixed-frame origin;
4. View distance: Orbit Distance ≈ 1.5–2.0 m for an RViz-like close full-body view (> 5 m shrinks the robot to a tiny center blob);
5. Health signal: ~3 s after startup the node logs FM: ... frames=N and transformHasProblems(...)=0 — meshes correctly bound to TF.

MoveIt2 motion — five essential modes, one tool

Design intent. Robot motion through MoveIt is, at its core, only five operations: set joints absolutely, nudge joints relatively, place the end-effector at an absolute pose, move it by a relative delta, or execute a pre-planned trajectory. Rather than a growing zoo of named tools, dsh-ros2 abstracts them into one tool moveit_move with a mode parameter — so the interface stays small, predictable and scriptable, and works with any MoveIt package (it reads the SRDF and speaks only standard moveit_msgs).

moveit_move {mode: "joint_abs", group: "right_arm", joints: "right_shoulder_roll:=0.5"}
moveit_move {mode: "joint_rel", group: "right_arm", deltaJoints: "right_elbow_pitch:=-0.2"}
moveit_move {mode: "pose_rel",  group: "right_arm", deltaPose: "0.05 0 0 0 0 0"}
moveit_move {mode: "pose_abs",  group: "right_arm", pose: "0.5 0 0.8 0 0 0"}
moveit_move {mode: "trajectory", group: "right_arm", trajectory: "/tmp/traj.json"}

How it works. moveit_discover/moveit_status tell you what you can move (groups, joints, EE link from the SRDF chain tip, online state); moveit_move turns any mode into a standard MoveGroup goal (/move_action), executes via /execute_trajectory, and — with planOnly + trajectoryOut — saves the planned trajectory so mode: "trajectory" can run it later (plan/execute separation). Nothing is bound to a specific MoveIt package; only the SRDF path matters (auto-resolved by package scan, or explicit srdf/package).

Safety framework

Design intent. A two-tier strategy that keeps real-time and semantic judgment apart (full contract: docs/safety-handover.md, the handover doc for downstream robot-adaptation agents — generic framework/interfaces belong here, body-specific data sources/schemes/algorithms belong downstream):

tool-layer safety gate (pre-execution: /safety/state LOCKED; monitor-down per safetyStrict)
  → reactive monitors (in-execution: motion tracking/stall + hysteresis,
    joint-feedback loss, watchdog, optional torque; detect at control frequency,
    response budget ≤100 ms)
  → VLM semantic arbitration (post plan-change / anomaly, seconds — pulled up only when needed)
  → human arbitration (non-safe verdicts always escalate; human-gated unlock)

• safety_monitor node (dsh_ros2_safety package): subscribes the joint feedback (+ optional commanded stream / torque stream), runs the checkers on a control_frequency timer, and latches LOCKED on any CRITICAL event — the latch persists until a human unlocks (no auto-reset into the same danger). Publishes /safety/state (transient-local), /safety/event, /safety/heartbeat, /safety/lock_active; services /safety/get_state, /safety/unlock, /safety/set_lock.
• Latched lock, non-fatal never locks: any CRITICAL event latches LOCKED (the latch persists until a human unlocks — no auto-reset into the same danger); watchdog separates critical (down → lock) from observed (down → WARNING only); single-frame noise is filtered by M-of-K hysteresis; a WARNING never latches. Tool-layer fail-closed on monitor-down is per safetyStrict: 'reject' (default 'warn' proceeds with a warning).
• Per-robot registration: robot_register writes a generic safety section (URDF-derived velocity/effort limits) and auto-launches the monitor; robot_profile.py safety set <key> <json> updates any threshold/topic/list (schema-validated). Torque checking auto-disables when no effort feedback exists. Reserved layers (interfaces registered, not implemented): the geometric pre-check (joint-limit / velocity / FK self-collision on the command path — motion.max_velocity / max_acceleration), the computed-torque feedforward input (torque.feedforward_topic), YOLO-style lightweight triggers, and the non-ROS estop path (estop).
• Tool-layer gate: moveit_move consults /safety/state before executing — LOCKED always rejected (SAFETY_LOCKED); if the monitor is unreachable, safetyStrict: 'reject' (fail-closed) or 'warn' (default) proceed-with- warning. robot_safety_arbitrate runs the fixed-format VLM arbitration and flags any non-safe verdict for human arbitration.
• Forensics: a ring buffer of joint/torque samples is dumped to forensics.dump_dir on every CRITICAL lock, for post-hoc / VLM diagnosis.

# build the safety package (same colcon workspace as vlm/ and offscreen/)
ln -s <repo>/safety /tmp/vlm_ws/src/dsh_ros2_safety
cd /tmp/vlm_ws && colcon build --symlink-install && source install/setup.bash

The safety_core pure logic ships with a fault-injection self test (python3 safety/scripts/safety_core.py --selftest, 12 scenarios) — no ROS2 needed to verify the state machine.

Robot profiles & communication topology

Design intent. A robot's body (URDF links/joints, cameras, MoveIt groups, zero-pose semantics) and its comms graph are costly to rediscover every time. The plugin persists them as a structured profile (~/.dsh-ros2/robots/<name>.yaml): register once on first contact, then load instantly forever after. For the comms graph, full verbosity does not scale (a complex robot has hundreds of topics/services), so the design is a trade-off: an aggregate snapshot (light node/topic/service lists) plus progressive learning of only the important nodes as you actually work with them.

Two bundled skills complete the workflow: robot-registration (first-contact flow: ask name/URDF → collect → register → verify, plus the topology baseline snapshot) and robot-retrieval (load the profile instantly and bring up renders/diagnostics/motion from it, including the learned topology and zero-pose semantics).

How it works. Everything is plain structured YAML under ~/.dsh-ros2/ — robots/<name>.yaml (body + topology) and zero-pose.yaml (calibration). robot_register snapshots the body; robot_topology snapshot snapshots the aggregate layer; robot_topology learn appends one node at a time (idempotent merge) as you discover what matters. robot_load and robot_topology show read them back instantly — one call instead of N discovery calls.

Bundled skills

Configuration

rosLogDir also covers ROS2 Python CLIs spawned by tools (topic echo/pub, ros2 run); additionally runCommand auto-falls back to a writable dir when ~/.ros/log is not writable.

Project layout

dsh-ros2/
├── src/                  # DSH plugin core (TypeScript)
│   ├── index.ts          # entry: registers 51 tools + 4 skills
│   ├── tools.ts          # tool definitions (L1/L2/L3 params & command mapping)
│   ├── vision.ts         # L4 vision tools (snapshot / analyze / topics)
│   ├── gui.ts            # L3 GUI lifecycle & interaction
│   ├── skill.ts          # 4 skills: diagnostics / vision / registration / retrieval
│   ├── runner.ts         # command runner (timeout/log/approval/background-job seams)
│   └── config.ts         # configuration read & validation
├── vlm/                  # ROS2 package dsh_ros2_vlm (Python): vlm_node / vision_bringup / vlm_bridge_node / image_snapshot / vlm_call / vlm_bridge_call
├── offscreen/            # ROS2 package dsh_ros2_rviz_offscreen (C++): rviz_offscreen_node (OGRE offscreen render → /rviz/scene)
├── safety/               # ROS2 package dsh_ros2_safety (Python): safety_monitor node + safety_core (pure logic, --selftest) + safety_vlm_arbitrate + SafetyState/Event msgs + Unlock/SetLock srvs
├── docs/                 # architecture.md · compatibility.md · safety.md / safety-handover.md / safety-todo.md / safety-gpt-review.md · test-robot-state-vision.md / test-gpu-passthrough.md · screenshots
├── tests/                # vitest (113 cases; CLI outputs mocked)
├── .github/workflows/    # CI: Node 22/24 → typecheck/test/build/pack validation
├── PUBLISH.md            # open-source publishing checklist (GitHub + npm + DSH community)
└── CHANGELOG.md          # version history (Keep a Changelog)

Troubleshooting / FAQ

• ~/.ros/log Permission denied: ROS2 cannot write its log dir. Set rosLogDir (e.g. /tmp/ros-log); runCommand also falls back automatically.
• A flood of RTPS_TRANSPORT_SHM / FastDDS SHM warnings in stderr: harmless noise when SHM transport is unavailable (common in containers/restricted environments); tools drop it by default.
• ros2 topic echo returns empty: check ros2_topic_info -v for publisher count and QoS; sample transient-local topics with --qos-durability transient_local.
• Offscreen render: "all parts piled at origin": URDF link names do not match TF frame names (see mesh essentials #3); check the node log FM transformHasProblems(<link>)=1 first.
• RobotModel shows no meshes (Links empty): on Jazzy you must use Description Source/Topic; the legacy Robot Description: does nothing.
• Could not load resource ... Unable to open file: mesh paths need absolute paths or a file:// prefix.
• Description missing after the publisher exits: the URDF publisher must stay alive (transient-local only re-sends once to late subscribers; gone when the process exits).
• vision_bringup missed some image topics (2/4 measured): fixed — it now refreshes discovery every --refresh seconds (default 10), auto-spawning bridges for topics that appear later and stopping bridges for topics that disappear.

Development

pnpm install
pnpm run typecheck   # tsc --noEmit
pnpm run test        # vitest (113 cases; CLI outputs mocked)
pnpm run build       # tsc -> lib/ + lib/types/

CI (.github/workflows/ci.yml): on push to main / PRs, runs typecheck/test/build on Node 22 and 24, and validates that the pnpm pack artifact contains the patch layer (cordis.patch.yml) and the build output.

Release workflow (npm & GitHub Releases): see PUBLISH.md.

Roadmap

• vision_bringup polling/refresh discovery (auto-bridge late topics, stop bridges for gone topics);
• Zero-pose semantics: generic calibration flow (ros2_zero_pose_semantics, render + VLM + user confirm, 3-axis combos) linked into robot profiles;
• npm publishing (pnpm publish --access public, requires npm login);
• More ROS2 distros (Humble / Rolling) compatibility validation.

Docs

Contributing

Issues and PRs welcome (in Chinese or English). Please keep pnpm run typecheck && pnpm run test && pnpm run build green and update the relevant docs.

Acknowledgments

• DeepSeek Harness — plugin host framework;
• ROS2 / RViz2 community — rendering and toolchain foundations.

License

MIT