Why PX4’s 10-second link-loss failsafe fires late

The PX4 parameter reference is short about it. COM_DL_LOSS_T is the time after which, with no data link, the GCS connection loss failsafe triggers.⁵ The default is 10 s. Set NAV_DLL_ACT to Hold, cut the link, and you would expect the vehicle to stop 10 s after the last ground station heartbeat reached it.

It stops later. On the tracked ground robot in our example it stopped 11.9 s after the last heartbeat. The parameter is not wrong, and nothing on the vehicle was slow. The timer starts somewhere other than where most people think.

Where the timer starts

The data link check lives in the commander. It never looks at heartbeats. It looks at telemetry_status, the uORB topic each MAVLink instance publishes, and it restarts the clock every time a report still says a ground station heartbeat is present:¹

if (telemetry.heartbeat_type_gcs) {
	// Initial connection or recovery from data link lost
	...
	_datalink_last_heartbeat_gcs = telemetry.timestamp;
}

Note the right-hand side: telemetry.timestamp, the time the status report was published, not the time the heartbeat arrived. The loss test a few lines later measures from that value:

// GCS data link loss failsafe
if (!_vehicle_status.gcs_connection_lost) {
	if ((_datalink_last_heartbeat_gcs != 0)
	    && hrt_elapsed_time(&_datalink_last_heartbeat_gcs) > (_param_com_dl_loss_t.get() * 1_s)) {

		_vehicle_status.gcs_connection_lost = true;
		...
		mavlink_log_info(&_mavlink_log_pub, "Connection to ground station lost\t");

So the question becomes: after the last heartbeat arrives, how long do status reports keep saying one is present?

How long the flag stays true

The MAVLink receiver owns the flag. It sets heartbeat_type_gcs to true while the last GCS heartbeat is less than HEARTBEAT_TIMEOUT_US old, which is 2,500,000 µs.⁴ It only re-evaluates the flag inside CheckHeartbeats(), and that runs in two situations:²

• whenever a heartbeat arrives from a ground station, or from another component of the vehicle’s own system such as the companion computer, as a forced check;
• otherwise, once 1.25 s have passed since the last check (TIMEOUT / 2).

if ((t >= _last_heartbeat_check + (TIMEOUT / 2)) || force) {
	telemetry_status_s &tstatus = _mavlink.telemetry_status();
	...
	tstatus.heartbeat_type_gcs = (t <= TIMEOUT + _heartbeat_type_gcs);

Between checks the flag keeps its last value, and the status topic goes out at least once a second whether or not anything changed.³ Every one of those reports that still carries true moves the commander’s start line forward.

Put together: the flag turns false at the first check more than 2.5 s after the last heartbeat, which is up to 1.25 s later still. The last report that carried it went out up to a second before that. Add COM_DL_LOSS_T on top. With the default of 10 s, the failsafe lands somewhere between about 11.5 s and 13.75 s after the last heartbeat, depending on how other heartbeats and the 1 Hz publications happen to fall.

A worked example

This is INC-0143 from the Foxborne example dataset. It is an example, not a field report. A tracked ground robot running PX4 v1.16.0 crosses an obstacle belt at 1.25 m/s. Its mesh radio sits on the companion computer’s USB bus, and the companion forwards ground station traffic to PX4 over TELEM2. The companion shares the vehicle’s MAVLink system ID and sends its own heartbeat to PX4 once a second.

1. 15:41:02.412. The last GCS heartbeat reaches PX4 on TELEM2. The radio drops off the USB bus 192 ms later.
2. 15:41:04.286. The companion’s heartbeat at 04.284 forces a check. The GCS heartbeat is 1.872 s old, under 2.5 s, so the report published at 04.286 still says true. It is the last one that will.
3. 15:41:05.286. The next companion heartbeat, at 05.284, forces another check. Now the GCS heartbeat is 2.872 s old, over 2.5 s, and the report published at 05.286 says false.
4. 15:41:14.286. Ten seconds after the report at 04.286, COM_DL_LOSS_T runs out.
5. 15:41:14.293. On its next 10 ms cycle the commander sets gcs_connection_lost. With NAV_DLL_ACT = 1 the failsafe selects Hold, which is never delayed by COM_FAIL_ACT_T, and the log shows “GCS connection loss: switching to Hold”. The vehicle is stopped 1.0 s later.

From the last heartbeat to the Hold: 11.881 s. From the last report that carried the heartbeat: 10.007 s. Both numbers are correct. Only one of them matches the parameter.

Read it off your own log

You can check this on any flight log that has a link loss in it. Export the status topics with pyulog:⁶

ulog2csv -m telemetry_status,vehicle_status,failsafe_flags 15_37_12.ulg
# writes 15_37_12_telemetry_status_1.csv for the TELEM2 instance, and so on

1. In the telemetry_status file for the link that carries your ground station, find the last row where heartbeat_type_gcs is 1. Its timestamp is the commander’s start line.
2. In failsafe_flags, find the first row where gcs_connection_lost is 1.
3. The difference should be COM_DL_LOSS_T plus at most one commander cycle. If it is not, something else is going on, and that is worth knowing too.

Why it matters in an investigation

A reviewer who compares the Hold to the last heartbeat and sees 11.9 s instead of 10 s will reasonably ask whether the failsafe misbehaved, or whether the heartbeat time is wrong, or whether the clocks are misaligned. None of those is true here, and the report should say so up front, with the status reports that prove it.