Skip to content

Network

Troubleshooting Banner

Jetson Xavier NX

ubuntu 24.04

Network Troubleshooting#

Socket Receive Buffer Size#

Cause: Kernel UDP socket buffer limits are too low for CycloneDDS (Linux default is ~212 KB).

Important

📚 All the following instructions must be applied on the host machine, not inside a container.

CycloneDDS requires significantly larger UDP socket receive buffers than the Linux default (~212 KB). Without this, node initialization fails with: failed to increase socket receive buffer size

For checking the current values, please run:

sysctl net.core.rmem_max
sysctl net.core.rmem_default
sysctl net.core.wmem_max
sysctl net.core.wmem_default

If values are set to default (~212 KB) you will have to update them, please follow these instructions Setting rmem wmem values

ros2 topic list returns only /parameter_events and /rosout#

Checklist:

# 1. Verify containers are running
docker ps

# 2. Verify environment variables match containers
echo $ROS_DOMAIN_ID         # must be 0
echo $RMW_IMPLEMENTATION    # must be rmw_cyclonedds_cpp
echo $CYCLONEDDS_URI        # must point to a valid file

# 3. Verify the config file exists and is readable
cat $CYCLONEDDS_URI

# 4. Check if containers can see their own topics
docker exec micipsa_description bash -c \
  "source /ros2_ws/install/setup.bash && ros2 topic list"

# 5. Verify you are on the same subnet as the robot
ip addr show   # look for 192.168.0.x
ping 192.168.X.X  # ping the robot

Topics appear briefly then disappear#

Cause: OS send buffers are too small. Publishers are dropping messages before they leave the machine.

Check current values of rmem and wmem:

Fix: 

cat /proc/sys/net/core/wmem_max       # should be 67108864
cat /proc/sys/net/core/wmem_default   # should be 67108864
cat /proc/sys/net/core/rmem_max       # should be 67108864
cat /proc/sys/net/core/rmem_default   # should be 67108864

If the values are wrong, follow setting rmem wmem values instructions.

RViz2 crashes on startup#

failed to increase socket send buffer size to at least 10485760 bytes
terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
Aborted (core dumped)

Cause: wmem_max on the host machine is below the SocketSendBufferSize min="10MB" requested in cyclonedds.xml.

Fix: apply the OS buffer fix (setting rmem wmem values) on the host machine, then relaunch RViz2.

Containers see each other but Host Machine sees nothing#

Cause: either the host machine cyclonedds.xml is missing, uses the wrong interface, or environment variables are not set.

Fix:

Checking environment variables

Checking CycloneDDS XML file

Verify the interface name matches what ip addr show reports

grep "NetworkInterface" /etc/cyclonedds.xml
ip addr show | grep "state UP"

CYCLONEDDS_URI is set but CycloneDDS ignores it#

Cause: the file:// prefix must be followed by an absolute path with three slashes total:

export CYCLONEDDS_URI=file:///etc/cyclonedds.xml
#                          ^^^
#             three slashes = absolute path

Multicast not working on loopback#

Symptom: inter-container communication is unreliable even with name="lo".

Check: 

ip link show lo
# Should include MULTICAST in flags:
# <LOOPBACK,UP,LOWER_UP,MULTICAST>

Fix if MULTICAST is missing:

sudo ip link set lo multicast on

Host publishes topics but robot intermittently does not receive them#

Symptom: ros2 topic pub or a teleop node on the host appears to publish successfully, the topic is visible in ros2 topic list and ros2 topic echo shows messages but the robot does not respond. The failure is not consistent: it works on some runs and fails on others with no apparent change in configuration.

Cause: loopback address leaking into cross-machine DDS discovery.

The robot's cyclonedds.xml lists both lo and wlan0. Because lo has multicast="true", CycloneDDS sends discovery announcements on loopback as well as WiFi. These loopback announcements advertise 127.0.0.1 as a valid unicast endpoint. The host machine receives both 127.0.0.1 and the robot's actual WiFi IP (192.168.0.15) via multicast discovery, and selects one to use for all subsequent unicast data delivery.

The selection depends on which advertisement arrives first, a race condition on WiFi where multicast packet ordering is non-deterministic. When 127.0.0.1 wins the race, the host sends all published data to its own loopback interface. The robot never receives it. No error is raised on either side.

Confirm Issue with tcpdump:

Run both simultaneously, then publish a test message from the host.

On the Jetson: 

sudo tcpdump -i lo udp -n

On the host: 

sudo tcpdump -i lo udp -n

Publish from the host: 

ros2 topic pub --once /micipsa_base_controller/cmd_vel \
  geometry_msgs/msg/TwistStamped \
  "{header: {stamp: now, frame_id: ''}, twist: {linear: {x: 1.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}}"

If the bug is active, the host tcpdump shows UDP packets between 192.168.0.x and 192.168.0.x on loopback (the host sending to itself), while the Jetson tcpdump shows no new unicast traffic at the time of the publish .

Because the failure is a race condition, it may not reproduce on every attempt. A single successful publish does not rule out the issue.

Fix: add an explicit <Peers> entry to the host machine's cyclonedds.xml:

<Discovery>
  <ParticipantIndex>auto</ParticipantIndex>
  <Peers>
    <Peer address="192.168.0.X"/>
  </Peers>
</Discovery>

This bypasses multicast address selection entirely, forcing all unicast data from the host to go directly to the robot's WiFi IP. See Peers for the full explanation and options for handling IP address changes.

Once changes made, make sure to source your ~/.bashrc on the host and restart micipsa stack on the robot/+.

Debugging Connectivity Checklist#

  • Are all machines on the same subnet? (ip addr show)
  • Are ROS_DOMAIN_ID and RMW_IMPLEMENTATION identical everywhere?
  • Does $CYCLONEDDS_URI point to an existing file on every machine?
  • Does each machine's xml name the correct interface? (ip addr show)
  • Are OS buffers tuned on every machine? (cat /proc/sys/net/core/wmem_max)
  • Is multicast enabled on loopback? (ip link show lo)
  • Can containers see their own topics? (docker exec ... ros2 topic list)
  • Is <Peers> set on the host machine pointing to the robot's WiFi IP?