IK Backend Benchmark

​

How it works

• Scenarios are drawn from SCENARIOS — one or more robots sharing a world (currently xarm6, xarm7, and dual_xarm6, a two-arm setup mirroring the dual-xarm6-planner-coordinator blueprint). Multi-robot scenarios are solved as joint multi-target solve_pose_targets calls. Each backend run builds its own fresh RoboPlanWorld — no shared mutable scene between backends.
• Targets are sampled by drawing joint configurations uniformly within limits for every robot, keeping only scene-wide collision-free ones, and mapping them through the world’s FK. Every target is reachable by construction, so failures measure solver behavior rather than unreachable goals.
• Fairness: all backends solve identical targets with an equivalent seed joint state, check_collision=True, and the same max_attempts.
• Latency is time.perf_counter around KinematicsSpec.solve, after warmup solves are discarded.
• Accuracy: every successful solution is independently re-verified by pushing it through the world’s FK and scoring against the target with compute_pose_error — the same metric for every backend.
• Resource usage: coarse current-RSS delta (/proc/self/statm) across each backend’s measured window; model construction happens during warmup.

​

Running

uv sync --extra manipulation --inexact

# Full benchmark (all scenarios, both backends):
uv run python misc/ik_benchmark/ik_backends.py --samples 200 --warmup 10 --output /tmp/ik.json

# Single scenario / backend, custom attempt budget:
uv run python misc/ik_benchmark/ik_backends.py --scenario dual_xarm6 --solver pink --max-attempts 3

# Sweep the attempt budget to trace the success-rate vs latency tradeoff:
for n in 1 2 5 10; do
  uv run python misc/ik_benchmark/ik_backends.py --max-attempts "$n" --output "/tmp/ik_a$n.json"
done

​

Extending

• New scenario: add a ScenarioSpec to SCENARIOS — a name plus a factory returning the RobotModelConfig list for one world (one entry per arm for multi-robot setups).
• New backend: add a SolverSpec to _solver_registry() — a name plus a constructor from a fresh WorldSpec to a KinematicsSpec. Backends that are world-agnostic (like Pink) may ignore the world; world-native backends (like OInK) take it as their solver.

​

Results