Design Overview: Operations Server

[NOTMOVETON]

Feature request

Operations Server: A generic executor for non-navigation tasks at some poses

Feature description

I propose a system designed to execute arbitrary non-navigation actions (e.g., toggling signal lights, setting manipulator poses, controlling onboard equipment) at specific positions or at current robot's position.

Currently, the WaypointFollower or Behavior Server is the standard tool for task execution, but it has limitations in this specific context, like: does not natively support defining multiple distinct actions at a single point or executing actions in parallel with motion without significant customization; doesnot have unified interface, doesnot have unified control node for BT (like FollowPath for example).

Possible solution:
The Operations Server acts as a generic service that allows executing various combinations of actions (even simultaneously) via a unified interface. Is servers as unified executor that is using Operation Plugins, each of them have same interface (like Controller Plugins do). The server itself should probably track the robots position and execute actions if some pose is achieved, but this still should be discussed in more details. The server itself should also be capable of pausing and resuming, in other words - take care of the non-navigation actions states. Also there should be some flag for blocking/non-blocking operations - need to discuiss in more details.

This feature allows for scenarios such as:

• Parallel execution: Driving while simultaneously performing an action (e.g., a lawnmower engaging its blades).
• Strict path adherence: Executing actions at precise points on a fixed route.
• Waypoint following with different actions: Executing actions at specific waypoints. Each waypoint can contain any set of actions.
• Complex state handling: Managing "True Pause" scenarios where pausing the robot also requires transitioning hardware to a safe state (e.g., stopping the rotor blades, not just the wheels).

Implementation considerations

To integrate this logic into the NAV2 ecosystem, the implementation relies on the following components:

1. Operations Server
A standalone server responsible for executing non-navigation tasks. It exposes a unified interface and can be used even without navigation or BT.

2. Custom Interfaces
A new .action messages types, that would be used by new Navigator plugins.

3. Navigator Plugins
A custom Navigator plugins each one for new behavior (follow path + make some operations along; follow waypoints or waypoint with actions; etc. ).

4. Behavior Tree Integration

• Unified Control Node: Like FollowPath node, but for actions.
• Pause and Resume Logic: The Operator Server and its plugins must support a "True Pause." This ensures that when the navigation is paused, the external equipment is also halted or moved to a safe state (e.g., stopping a cutting rotor), rather than just cancelling the goal. This functionality also already exist as BT nodes.

The logic for when to trigger currently implemented in the tree with special decorator, but this approach isn't stable and definitely not safe, so I think we should move the pose tracking directly to the Operations server and modify the logic.

I would provide more details later in following messages.

[Innomer]

While this feature seems interesting and could be helpful for overall robot orchestration, I'm not entirely convinced it requires introducing a dedicated Operations Server. Nav2 already uses Behavior Trees as the primary orchestration layer, and many of the described scenarios could be implemented using custom BT nodes combined with the existing ROS 2 action/service interfaces as you have mentioned.

Given the number of parameters involved (blocking vs non-blocking execution, pause/resume semantics, hardware state management, etc.), it feels like this logic might be better handled at the Behavior Tree level rather than through another core server abstraction. Using constructs like Parallel nodes, decorators, or custom BT nodes that monitor robot pose could allow similar behavior while keeping the coordination logic within the BT, which is already intended to manage task sequencing and execution.

If anything, this could also make for a useful tutorial or example showing how to implement such patterns using existing BT mechanisms.

You mentioned that the current approach using decorators for pose-triggered actions is not stable and definitely not safe. Could you elaborate on the specific issues you've encountered with that approach? It would be helpful to better understand what limitations you see there.

[NOTMOVETON]

First of all, thank you for your comment and your thoughts. Here are some links that I think would be helpful to review: the discussion about the tutorial on this topic and the tutorial itself with the source code.

A detailed explanation of why relying solely on BT nodes is insufficient is provided in the tutorial, but briefly:

• Poor scalability. Imagine having new BT node for every new controller plugin configuration you add. Same here.
• Difficulty in maintaining a unified interface; each action would end up having its own custom interface.
• The inability to execute actions at positions that only become known at runtime, just like the path itself.

Given the number of parameters involved (blocking vs non-blocking execution, pause/resume semantics, hardware state management, etc.), it feels like this logic might be better handled at the Behavior Tree level rather than through another core server abstraction. Using constructs like Parallel nodes, decorators, or custom BT nodes that monitor robot pose could allow similar behavior while keeping the coordination logic within the BT, which is already intended to manage task sequencing and execution.

Could you provide a specific example of how you would structure this?

You mentioned that the current approach using decorators for pose-triggered actions is not stable and definitely not safe. Could you elaborate on the specific issues you've encountered with that approach? It would be helpful to better understand what limitations you see there.

Yes, this is related to ReactiveSequence and the fact that it halts child nodes that are no longer being ticked. This situation is also described in the tutorial. Even though this is core BT logic, it causes something similar to a data race when following a long path. Furthermore, it is tied to the issue that "instant" or "non-instant" actions actually turn out to be "short non-instant" (e.g., taking 3 ticks) and "long non-instant" (taking many ticks) in practice because of the decorator. Also it is better to move the tracking function of the operations execution in server itself and provide specific tracking plugins (like goal checker in Controller Server) and implement a tree with different structure to allow both blocking and non blocking execution change at runtime. By the way this would also require custom node that reads feedback from the Operations Server, but i think this approach would be more cleaner than current implementation.

I invite you to take a look at the tutorial I just finished writing yesterday, as well as the discussion with Steve, to get a deeper understanding of the context and the specific problem we are solving.

I would be glad to hear your thoughts and see your example :)

[NOTMOVETON]

@SteveMacenski
Actually there is not much infomation here. I would try to come up with the design and questions worth to discuss by the end of the week.

For now here is a list of abstract topics I would like to discuss:

• What does the problem sound like? What are we solving and what is this functionality for? My initial problem was that we have a path, at some points along the path the robot must perform an action, the action itself can be blocking - the robot cannot move while it is performing it, and non-blocking - instantaneous.

• is this functionality necessary in NAV2? This is a pretty deep topic, and perhaps doing it as part of navigation can be controversial. Right now, this idea looks like an improved waypoint follower + sort of controller server. Maybe it makes sense to redo waypoint follower server?

[SteveMacenski]

Sorry for the delay but I'm glad we chatted on the call this morning before I got to this in earnest.

To @Innomer's point, maybe the behavior tree can do some portion of this, but I think @NOTMOVETON is mentioning also that we need a navigator to take in a precomputed path / operations info with these operations to execute as a key part of this. Not just the server, but the full design of a new class of navigation task that uses pre-computed missions/paths and then wants to do "stuff" with it. So I think there's something to do this that we should definitely explore -- whether it looks and feels the same way as the initial proposal or not.

A couple of ways to think about this, I think at least:

1. We have a 'Operations Navigator' which takes in some file / name of an operations file. We use a BT node via an Operations Server to extract the path from this file to pass into the blackboard for path tracking. Optionally: the server or the BT tracks progress and triggers behaviors. A benefit is that we can use the server internal to the BT to parse the file.
2. We have a 'Operations Navigator' which is already given a pre-computed path and operations to perform. Then the BT just needs to track this path, likely track the current pose and nearest path points (which should be especially easy now that we have TrackingFeedback from the controller server). Then either streaming the nearest poses or the operations to trigger to the server. A drawback would be parsing the file beforehand from the autonomy / client application.

I generally believe that we should have semantically useful behavior tree nodes / ROS interfaces to particular things being done (i.e. flip on an LED, start a mower blade, etc) as both being strongly typed (a ROS important design principle) and meaningful for a user to read to develop behavior tree nodes without worrying about the details under the hood of ToggleLED.

With that said, ToggleLED could internally as an implementation detail implement a more abstract message type to interact with some server, which would solve one of those issues and leave it up to a user about if they want to work with ergonomic BT nodes or super generalized APIs.

The issue that arises if we want to execute arbitrary operations at arbitrary positions is that we need to be able to encode that information in some way that we can trigger those operations at any place. I can see where @NOTMOVETON comes up with the completely abstract design message so that we can store and send any operation anytime, anyplace. I don't love that idea as it breaks having strongly-typed interfaces, but I suppose if you had an application with legitimately 100+ possible arbitrary operations, it may be the most scalable solution. Also we would need some abstract, generalized way to store operations information as the annotations on the path regardless -- but it doesn't necessarily mean that's how the Actions / Services are called. I think that's a key insight.

I don't think most applications though have 100+ and we may be able to say that it is bounded within reason to be able to have unique action / service .srv and .action files for each. The design challenge then becomes how to store the operations in the recorded file and how to trigger the appropriate action when its time comes up. So, how do we store the arbitrary information in the Path to call, and how do we translate that into the Action/Service call that is strongly-typed for that operation server's task.

I don't have the immediate answer to that, but brainstorming:

• I don't think in C++ even if we have the string "nav2_msgs::action::MyAction" we can actually translate that into an object instance or type to create the action client or message instance. If we can, then it should be easy to populate the request fields. Using rosidl in Python3 it appears that we can, but BT.CPP nor the servers are python. I suppose we try to bind python to C++ in the BT node, but that's getting hacky.
• If we say, for instance, that we're OK creating unique actions and services for each operation, then I suppose we could also say that its reasonable to create unique behavior tree nodes for each as well. If that's the case, the type is known to us. We would just need to match an operation name to an action name (easy) and populate the request fields from the stored information. If we did that, we'd probably want to name all of the BT nodes that use this pattern with a prefix or suffix to make it clear that they implement the Operations API.
• The operations stored along the path to execute could also be stored as serialized json/yaml or something to be more ergonomic rather than a vector of all base types if we do something like above so we know how each operation translates into its request fields. It only works when we have strongly typed interfaces I think.

---

The server itself should probably track the robots position and execute actions if some pose is achieved, but this still should be discussed in more details.

I believe in the other thread you have the behavior tree itself doing this. Which do you propose?

I think the BT XML would be sensible to track the pose, or at least get the tracking information from the BT port for TrackingFeedback that we now have.

The server itself should also be capable of pausing and resuming, in other words - take care of the non-navigation actions states. Also there should be some flag for blocking/non-blocking operations - need to discuiss in more details.
Pause and Resume Logic: The Operator Server and its plugins must support a "True Pause." This ensures that when the navigation is paused, the external equipment is also halted or moved to a safe state (e.g., stopping a cutting rotor), rather than just cancelling the goal. This functionality also already exist as BT nodes.

Can you expand on this front? This seems important but I don't have an immediate intuition or understand the expansive uses of this.