Robot enters lift without session when its ongoing task was cancelled and assigned with another task

[cwrx777]

open-rmf release: iron 20231229

Issue summary: robot does enters lift without having lift session if while waiting for lift, robot current task is cancelled and assigned with another task.

Building yaml: hotel.building.yaml
Difference with original building yaml:

1. remove lane to Lift2. This is so that only Lift1 is used.
2. use unidirectional lanes to enter and exit Lift1
   

Robot starting position is set by using spawn_entity.launch.xml and robotplacer.launch.xml
Task is dispatched by using this modified dispatch_patrol.py

lift_session_issue_20241025.mp4

Initial position:
tinyBot_1 is at Lift_LWP_L1 and tinyBot_2 is at Lift1_LWP_L2.
First, send tinyBot_2 to a location at L1.

ros2 run rmf_demos_tasks dispatch_patrol -F tinyRobot -R tinyBot_2 -p restaurant --use_sim_time -id task_1;

Then, a moment later, send tinyBot_1 to a location on L2. This is to make tinyBot_1 to keep requesting for the lift.

sleep 5;
ros2 run rmf_demos_tasks dispatch_patrol -F tinyRobot -R tinyBot_1 -p L2_room15 --use_sim_time -id task_2;

(refer to video at 00:25), when tinyBot_1 reach L1 and lift door start to open, cancel tinyBot_2's task and send it to another location on L1.

ros2 run rmf_demos_tasks cancel_task -id task_2;
ros2 run rmf_demos_tasks dispatch_patrol -F tinyRobot -R tinyBot_1 -p restaurant --use_sim_time -id task_3;

Observation:
tinyBot_2 enters lift while tinyBot_1 is still inside.

[mxgrey]

Thanks for the detailed report. I've opened an issue ticket for this and we'll try to reproduce and then debug it.

It looks like you're using a release from December of last year, and several race conditions related to lifts and task cancellation have been put in since then, so I'll start by doing an A/B test of the scenario with the current main branch vs the one you mentioned.

[mxgrey]

Even without negotiation, it is not supposed to be possible for the robot to try to enter the lift without having the lift session locked in. The lift session is a totally separate mechanism from the traffic negotiation.

In my experiment I saw that the robot tried to drive into the lift without having the lift session locked, and this was the key problem. You can check this by running ros2 topic echo lift_states and looking at session_id. If the robot is trying to enter the lift when the session_id is not its name or when the lift is not at the correct floor with the doors open then the robot is misbehaving.

This problem was able to happen whether or not another robot was already in the lift, but it only happens when the robot is starting from the waypoint that immediately precedes the lift. Notice how in your latest video the robot is starting two waypoints away from the lift and then subsequently does not try to enter the lift until it has the lift session locked in.

[cwrx777]

Hi @mxgrey,
Do you manage to come up with the fix?

[mxgrey]

I've made enough progress to figure out that the problem isn't what I thought it was.

It turns out that the plans being generated are correct and include an event to begin a lift session. But for some reason that event is being completely skipped while the plan is being executed. I suspect there's a subtle bug when ExecutePlan generates the event sequence from the plan, but I haven't fully confirmed this yet. Since I haven't seen this kind of problem happen before I don't have a good estimate for when I'll have a solution.

In the meantime, to make sure this doesn't happen I recommend taking two measures:

• Use a single mutex group across all lanes and vertices inside a lift (on all floors) as well as outside of the lift (on all floors) up to the point where you want the robot to wait for the lift as well as up to the point where a robot exiting the lift will no longer be in the way of a robot that wants to enter.
• Make sure you arrange these waypoints and lanes in a way that the robots will not have conflicts with each other or get deadlocked by the mutex.

It's extra work but I strongly recommend doing this for any deployments to ensure that lift traffic is infallible. Otherwise you can experience very significant disruptions whenever conflicts happen near lifts. Even when I fix the bug that's currently happening, it will be beneficial for you to have a rigorous mutex group with careful traffic flow around each lift.

[cwrx777]

@mxgrey
Thank you for taking the time to investigate this issue.

[mxgrey]

The bug has been found and fixed here: open-rmf/rmf_ros2#393

There are exactly two ways that the bug can occur:

1. A task gets cancelled while the robot is waiting to use a lift, and a new task is assigned (such as the finishing task) which tells the robot to use the same lift right away.
2. A negotiation or replan happens while the robot is waiting to use a lift, and the new plan tells the robot to use the same lift right away

I believe all the bugged cases that you've discovered match one of those categories, so the fix should cover everything.

That said, I would still recommend making sure that robots can always enter and exit lifts without a risk of conflicts occurring (e.g. by using mutex groups) since negotiating may still cause significant disruption during the somewhat brittle process of getting in and out of lifts.