Manipulation

Motion planning and teleoperation for robotic manipulators. RoboPlan provides the default world and native path planner.

Quick Start

Recent addition: the A-750 keyboard teleop blueprint is now available via:

dimos run keyboard-teleop-a750

Keyboard Teleop (single command)

Each blueprint launches the full stack — keyboard UI, mock controller, IK solver, and Drake visualization:

dimos run keyboard-teleop-a750    # A-750 6-DOF
dimos run keyboard-teleop-piper   # Piper 6-DOF
dimos run keyboard-teleop-xarm6   # XArm6 6-DOF
dimos run keyboard-teleop-xarm7   # XArm7 7-DOF

Open the Meshcat URL printed in the terminal (default http://localhost:7000) to see the robot. Keyboard controls:

Key	Action
W/S	+X/-X (forward/back)
A/D	+Y/-Y (left/right)
Q/E	+Z/-Z (up/down)
R/F	+Roll/-Roll
T/G	+Pitch/-Pitch
Y/H	+Yaw/-Yaw
ESC	Quit

Motion Planning (two terminals)

# Terminal 1: Mock coordinator
dimos run coordinator-mock

# Terminal 2: Planner with Drake visualization
dimos run xarm7-planner-coordinator

The IK default follows the planning world: RoboPlan worlds use bundled OInK, while other worlds use Pink. Select Pink explicitly to tune it with nested module config overrides:

dimos run xarm7-planner-coordinator \
  -o manipulationmodule.kinematics.backend=pink \
  -o manipulationmodule.kinematics.max_iterations=100 \
  -o manipulationmodule.kinematics.dt=0.02

For blueprints that instantiate PickAndPlaceModule, use the corresponding module prefix:

dimos run xarm-perception-sim \
  -o pickandplacemodule.kinematics.backend=pink

Then use the IPython client:

python -m dimos.manipulation.planning.examples.manipulation_client

skip

joints()                # Get current joints
plan([0.1] * 7)         # Plan to target
preview()               # Preview in Meshcat
execute()               # Execute via coordinator

Planning backend selection

Manipulation planning separates the world backend from the planner algorithm:

world_backend selects the robot/world/collision representation.
planner_name selects the path-planning algorithm.
Omitted kinematics selects the world-native IK backend when available and otherwise selects Pink.
Explicit kinematics.backend overrides the automatic selection. The legacy kinematics_name field remains an explicit compatibility shim.

dimos run xarm7-planner-coordinator

Select the legacy Drake world and generic RRT planner explicitly when needed:

dimos run xarm7-planner-coordinator \
  -o manipulationmodule.world_backend=drake \
  -o manipulationmodule.planner_name=rrt_connect

Valid combinations:

`world_backend`	`planner_name`	`kinematics.backend`	Status
`roboplan`	`roboplan`	omitted or `roboplan`	Default native RoboPlan world, planner, and OInK
`roboplan`	`roboplan`	`pink` or `jacobian`	Explicit generic IK override
`drake`	`rrt_connect`	omitted or `pink`	Legacy Drake world with Pink
`drake`	`rrt_connect`	`jacobian`	Legacy Jacobian IK
`drake`	`rrt_connect`	`drake_optimization`	Drake-only IK
`roboplan`	`rrt_connect`	omitted, `roboplan`, `pink`, or `jacobian`	Native or generic IK with generic RRT

Invalid combinations fail during startup instead of waiting for the first plan request. For example, planner_name=roboplan requires world_backend=roboplan, and kinematics.backend=drake_optimization requires world_backend=drake. Likewise, kinematics.backend=roboplan requires a RoboPlan world. Install the manipulation dependencies:

uv sync --extra manipulation --inexact

The manipulation extra includes RoboPlan via roboplan from PyPI. OInK is bundled as roboplan.optimal_ik; it has no separate install or capability probe. The --inexact flag preserves other extras already installed in your current environment. Safety behavior for unsupported RoboPlan features:

RoboPlan OInK accepts pose targets expressed in the world frame. Other target frames return an unsupported IK result.
OInK checks convergence against the requested position and orientation tolerances. When collision checking is enabled, the converged composite endpoint is checked in the RoboPlan world.
The initial OInK integration does not add a self-collision barrier. Intermediate numerical iterations are search states, not a safe executable path; the motion planner remains responsible for generating a collision-free trajectory.
Planning-critical unsupported inputs fail loudly before planning. Examples include unsupported obstacle geometry, unavailable robot loading APIs, or unavailable collision query APIs. RoboPlan worlds generate a minimal SRDF from the DimOS robot config, including configured collision-exclusion pairs.
Unverified non-critical query methods raise explicit NotImplementedError. In particular, signed minimum-distance semantics are not implemented for RoboPlan until a safe equivalent is verified.
Embedded Meshcat visualization requires a world implementing VisualizationSpec; use Viser or none with the RoboPlan backend.

Planning Visualization

Manipulation visualization is configured on ManipulationModuleConfig.visualization. It is independent from the global Rerun stream viewer in docs/usage/visualization.md. Backend choices:

meshcat: embedded Drake/Meshcat visualizer. The planning world must be created with embedded visualization enabled, so this is selected through the visualization config.
viser: in-process Viser visualizer. It renders pushed current robot state, target controls, transient preview ghosts, synchronized trajectory previews, and optional panel controls.
none: no manipulation planning visualization.

CLI example:

uv run dimos run xarm7-planner-coordinator \
  -o manipulationmodule.visualization.backend=viser

Blueprint example:

skip

from dimos.manipulation.manipulation_module import ManipulationModule, ManipulationModuleConfig

manipulation = ManipulationModule.blueprint(
    config=ManipulationModuleConfig(
        robots=[...],
        visualization={
            "backend": "viser",
            "host": "127.0.0.1",
            "port": 8095,
            "open_browser": True,
            "panel_enabled": True,  # default; set False for scene-only Viser
        },
    )
)

Viser support is included in the manipulation extra:

uv sync --extra manipulation --inexact

The Viser panel talks to the concrete ManipulationOperator bound into its VisualizationSession. GUI callbacks enqueue operations through that operator for target evaluation, planning, preview, execution, cancellation, reset, and clear-plan actions. The panel owns only target drafts, selection state, and callback generations; it does not touch WorldSpec, IK, planner objects, ManipulationModule, WorldMonitor, or live Drake contexts directly. External manipulation visualizers are initialized from a backend-neutral VisualizationSession after the planning world has added its robots. The session contains static PlanningSceneInfo metadata: world robot IDs, RobotModelConfig values, and resolved planning groups. Runtime joint state is then pushed through VisualizationStateFrame updates so renderers do not poll world/module state or own freshness policy. Embedded Meshcat visualization does not need extra setup because it observes the Drake world directly. Previews use the stored synchronized JointTrajectory from the generated plan. Viser projects the globally named trajectory into robot-local preview ghosts and plays the stored timestamped points directly; optional preview duration only scales the stored delays. Execute freshness is enforced by the manipulation module/operator immediately before dispatch, not by Viser-side telemetry snapshots.

Perception + Agent

# Coordinator + perception + manipulation + LLM agent (single command)
XARM7_IP=<ip> dimos run coordinator-xarm7 xarm-perception-agent

For a simulation walkthrough, see Agentic xArm simulation.

Architecture

KeyboardTeleopModule ──→ ControlCoordinator ──→ ManipulationModule
  (pygame UI)              (100Hz tick loop)      (WorldSpec backend)
       │                        │                       │
  TwistStamped           EEFTwistTask             RRT planner
  spatial EEF twist      (Pinocchio FK/IK)        JacobianIK
                               │                   DrakeWorld
                          JointState ────────────→ (visualization)

KeyboardTeleopModule — Pygame UI publishing routed spatial EEF twist intent
ControlCoordinator — 100Hz control loop with mock or real hardware adapters
ManipulationModule — world backend, optional visualization, RRT motion planning, obstacle management

Internally, planning code depends on WorldSpec for world, collision, and kinematics behavior. Meshcat preview and publishing are exposed separately through VisualizationSpec, so non-visual planning paths do not require a visualization backend.

Blueprints

Blueprint	Description
`keyboard-teleop-a750`	A750 6-DOF keyboard teleop with Drake viz
`keyboard-teleop-piper`	Piper 6-DOF keyboard teleop with Drake viz
`keyboard-teleop-xarm6`	XArm6 6-DOF keyboard teleop with Drake viz
`keyboard-teleop-xarm7`	XArm7 7-DOF keyboard teleop with Drake viz
`xarm6-planner-only`	XArm6 standalone planner (no coordinator)
`xarm7-planner-coordinator`	XArm7 planner with coordinator integration
`dual-xarm6-planner`	Dual XArm6 planning
`xarm-perception`	XArm7 + RealSense camera for perception
`xarm-perception-agent`	XArm7 perception + LLM agent
`xarm-perception-sim`	XArm7 simulation perception stack
`xarm-perception-sim-agent`	XArm7 simulation perception stack + LLM agent

Supported Robots

Robot	DOF	Teleop	Planning	Perception
A-750	6	Y	Y	—
Piper	6	Y	Y	—
XArm6	6	Y	Y	—
XArm7	7	Y	Y	Y

Adding a Custom Arm

guide is here

Key Files

File	Description
`manipulation_module.py`	Main module (RPC interface, state machine)
`robot/manipulators/common/blueprints.py`	Shared coordinator, planner, and task helpers
`robot/manipulators/a750/config.py`	A-750 model and hardware config
`robot/manipulators/a750/blueprints/teleop.py`	A-750 keyboard teleop blueprint
`robot/manipulators/piper/blueprints/basic.py`	Piper coordinator blueprint
`robot/manipulators/piper/blueprints/teleop.py`	Piper teleop blueprints
`robot/manipulators/xarm/blueprints/basic.py`	XArm coordinator and planner blueprints
`robot/manipulators/xarm/blueprints/perception.py`	XArm perception blueprint
`teleop/keyboard/keyboard_teleop_module.py`	Keyboard teleop module
`planning/world/drake_world.py`	Drake physics backend
`planning/planners/rrt_planner.py`	RRT-Connect motion planner

Getting Started

Installation

Usage

Capabilities

Platforms

Development

Quick Start

Keyboard Teleop (single command)

Motion Planning (two terminals)

Planning backend selection

Planning Visualization

Perception + Agent

Architecture

Blueprints

Supported Robots

Adding a Custom Arm

Key Files

​Quick Start

​Keyboard Teleop (single command)

​Motion Planning (two terminals)

​Planning backend selection

​Planning Visualization

​Perception + Agent

​Architecture

​Blueprints

​Supported Robots

​Adding a Custom Arm

​Key Files

Quick Start

Keyboard Teleop (single command)

Motion Planning (two terminals)

Planning backend selection

Planning Visualization

Perception + Agent

Architecture

Blueprints

Supported Robots

Adding a Custom Arm

Key Files