YOLO Pick-and-Place#

Detect cubes and dice with YOLO and pick and place them. The detector moves to a scan pose, visually servos over each detected object, picks it, and places it in the per-class drop box. See Example Applications for the shared manipulator setup. All commands run through mros.

Yolo pick-and-place

Yolo pick-and-place on simulation#

Yolo pick-and-place

Yolo pick-and-place on hardware-in-the-loop (HIL)#

Yolo pick-and-place

Yolo pick-and-place on real-world scenario#

  1. Open the Isaac Sim scene: ~/workspaces/movensys-simulation/<MANIPULATOR_MODEL>/8a_yolo_pick_and_place_simulation.usd

  2. Run the simulator bridge:

    mros ros2 launch movensys_manipulator_moveit_config sim_bridge.launch.py \
     simulator:=isaacsim use_sim_time:=true

  3. Launch the planner and service API:

    mros ros2 launch movensys_manipulator_moveit_config moveit.launch.py use_sim_time:=true

    mros ros2 launch movensys_manipulator_isaac_ros_config isaac_cumotion.launch.py use_sim_time:=true

  4. Launch the YOLO cube detector (the simulation scene contains cubes only):

    mros ros2 launch movensys_manipulator_perception yolo_cube_detector.launch.py use_sim_time:=true

  5. Execute the YOLO pick-and-place:

    mros ros2 launch movensys_manipulator_moveit_config yolo_pick_and_place.launch.py use_sim_time:=true

    (Pick/drop poses are loaded from config/<MANIPULATOR_MODEL>/yolo_trajectory.yaml.)

    Debug the detections (optional):

    ros2 run rqt_image_view rqt_image_view /yolo_cube_detector/debug_image

  1. Open the Isaac Sim scene: ~/workspaces/movensys-simulation/<MANIPULATOR_MODEL>/8b_yolo_pick_and_place_hil.usd

  2. Start WMX R2 for the manipulator (real WMX runtime) with use_sim_time:=true (see ~/workspaces/movensys_ws/src/wmx-r2/doc/launch_<MANIPULATOR_MODEL>_manipulator.md).

  3. Launch the planner and service API with use_sim_time:=true:

    mros ros2 launch movensys_manipulator_moveit_config moveit.launch.py use_sim_time:=true

    mros ros2 launch movensys_manipulator_isaac_ros_config isaac_cumotion.launch.py use_sim_time:=true

    Add rsp:=false if using Gazebo or ros2_control – both already publish /robot_description.

  4. Launch the YOLO cube and dice detector:

    mros ros2 launch movensys_manipulator_perception yolo_dice_and_cube_detector.launch.py use_sim_time:=true

  5. Execute the YOLO pick-and-place:

    mros ros2 launch movensys_manipulator_moveit_config yolo_pick_and_place.launch.py use_sim_time:=true

  1. Open the Isaac Sim scene: ~/workspaces/movensys-simulation/<MANIPULATOR_MODEL>/8c_yolo_pick_and_place_real.usd

  2. Start WMX R2 for the manipulator on the robot (see ~/workspaces/movensys_ws/src/wmx-r2/doc/launch_<MANIPULATOR_MODEL>_manipulator.md).

  3. Launch the planner and service API:

    mros ros2 launch movensys_manipulator_moveit_config moveit.launch.py

    mros ros2 launch movensys_manipulator_isaac_ros_config isaac_cumotion.launch.py

    Add rsp:=false if using Gazebo or ros2_control – both already publish /robot_description.

  4. Launch the YOLO cube and dice detector:

    mros ros2 launch movensys_manipulator_perception yolo_dice_and_cube_detector.launch.py

    Debug the detections (optional):

    ros2 run rqt_image_view rqt_image_view /yolo_dice_detector/debug_image
ros2 run rqt_image_view rqt_image_view /yolo_cube_detector/debug_image

  5. (optional) Execute the YOLO pick-and-place:

    mros ros2 launch movensys_manipulator_moveit_config yolo_pick_and_place.launch.py

Note

The YOLO pick-and-place is the perception layer used by the Robopoly Game (VLM / LLM), which adds voice and VLM/LLM control on top.