A telemetry gap is not a radio failure

A gap in the ground receive log proves one thing: nothing from the vehicle arrived. It does not say where the chain broke, and the chain is long. The flight controller, the serial link, the router on the companion computer, the radio, the radio path and the receiver can each go quiet.

It is tempting to blame the radio, which in the examples below is the one part that keeps no log. Here are three gaps with three different breaks, each an example from the Foxborne example dataset, not a field report. The other logs show which is which.

What a heartbeat can tell you

MAVLink treats a component as connected while its HEARTBEAT is regularly received, and as disconnected when a number of expected ones go missing.¹ MAVLink does not define the rate or the timeout; both depend on the channel.¹ On RF telemetry links, components typically send at 1 Hz and call a system disconnected after four or five missed heartbeats.¹

The spec also says a faulted component should stop broadcasting its heartbeat, and never send it from a thread unaware of the rest.¹ A heartbeat that keeps arriving is the sender’s own claim that it works.

So “disconnected” is a threshold someone chose. PX4 times out its per-link heartbeat flags at 2.5 s,³ and the example dataset flags a receive gap after 3 s. Neither number says why the heartbeats stopped.

Three gaps, one symptom

On the receive log alone, the three look alike: a steady run of 1 Hz receipts, then nothing.

• INC-0142, 6.96 s. A Q4 quad, UAS-04, on Route Iron. Nothing arrived from 14:32:03.912 to 14:32:10.874.
• INC-0143, 31.0 s. A T4 tracked robot, UGV-02, in the obstacle belt of Breach lane 2. Nothing arrived from 15:41:02.713 to 15:41:33.689.
• INC-0137, 18 min 18 s. A T4, UGV-05, patrolling the north side of a berm. Nothing arrived from 14:19:01.361 to 14:37:19.312.

The length does not sort them. It measures how long each break lasted, not where it was.

Ask PX4 who it still hears

Each MAVLink instance in PX4 publishes its own telemetry_status,⁵ with one flag per heartbeat type and a 2.5 s timeout.³ Two of the flags matter here:⁴

uint64 HEARTBEAT_TIMEOUT_US = 2500000       # Heartbeat timeout (tolerate missing 1 + jitter)

# Heartbeats per type
bool heartbeat_type_antenna_tracker         # MAV_TYPE_ANTENNA_TRACKER
bool heartbeat_type_gcs                     # MAV_TYPE_GCS
bool heartbeat_type_onboard_controller      # MAV_TYPE_ONBOARD_CONTROLLER

On TELEM2, the companion’s link, two of the examples split cleanly. In INC-0143 PX4 lost the ground station but still heard the companion. In INC-0142 it lost both:

INC-0142  14:32:06.510  boot + 1,135.224 s
heartbeat_type_gcs                  false
heartbeat_type_onboard_controller   false

INC-0143  15:41:05.286  boot + 241.775 s
heartbeat_type_gcs                  false
heartbeat_type_onboard_controller   true

A true heartbeat_type_onboard_controller means the serial link and the companion’s MAVLink side were alive, so the break sat beyond the companion. Losing both puts the break at or before the router. The router owns the flight controller’s UART: when it restarted, it logged “Opened UART [1]fc: /dev/ttyTHS1 baud=921600”.

In INC-0137, PX4 logged the same loss of the ground heartbeat on TELEM2, at 14:19:04.961. The flight log shows that the link was gone. It cannot show why.

Ask the companion what it saw

The companion’s journal says what happened on board, on a clock you can bound. Each line below shows __REALTIME_TIMESTAMP as UTC, the identifier and the message. In INC-0142, the kernel killed perception_node and systemd stopped the router 34 ms later:

14:32:04.118  kernel           Out of memory: Killed process 2213 (perception_node) total-vm:11873248kB, anon-rss:6823516kB, file-rss:10240kB, shmem-rss:0kB, UID:1001 pgtables:14720kB oom_score_adj:0
...
14:32:04.139  systemd          Stopping MAVLink router...
14:32:04.152  systemd          Stopped MAVLink router.
...
14:32:09.203  systemd          Started MAVLink router.
...
14:32:10.702  mavlink-routerd  Opened UART [1]fc: /dev/ttyTHS1 baud=921600

The router came back with the perception service at 14:32:09.203, and range control heard the vehicle again at 14:32:10.874. INC-0143 looks different. The kernel logged the radio leaving the USB bus, and the router kept running with nowhere to send:

15:41:02.604  kernel           usb 1-2.3: USB disconnect, device number 5
15:41:02.611  kernel           cdc_ether 1-2.3:1.0 usb0: unregister 'cdc_ether' usb-3610000.usb-2.3, CDC Ethernet Device
15:41:02.790  mavlink-routerd  Error sending udp packet (Network is unreachable)
...
15:41:31.802  kernel           usb 1-2.3: new high-speed USB device number 6 using tegra-xusb
15:41:32.214  kernel           cdc_ether 1-2.3:1.0 usb0: register 'cdc_ether' at usb-3610000.usb-2.3, CDC Ethernet Device, 02:1a:c7:40:5e:21
15:41:32.951  systemd-networkd usb0: Gained carrier

A stopped router and an unreachable network are different findings, and the journal tells them apart. The radio reappeared on the bus 29.2 s after it left, and PX4 heard the ground station again at 15:41:33.090.

In between, PX4 logged “GCS connection loss: switching to Hold” at 15:41:14.293, and the robot stopped. Why that came 11.9 s after the last heartbeat, with the link-loss timer set to 10 s, is the subject of a separate note.

In INC-0137 the companion kept working. The collector logged “uplink down, spooling locally” at 14:19:05.240 and “uplink restored, uploading 31.2 MB” at 14:37:20.202. Between them, the example’s journal holds no router stop and no USB disconnect.

Those lines waited for the link as well. The first was ingested at 14:37:20.200, 1,095 s after it was written, so the example shows event time and ingest time side by side.

Ask the map where the vehicle was

A radio path can fail while every part on the vehicle stays healthy, and nothing on board records why. The evidence is geometry: the straight line from the ground antenna to the vehicle’s antenna, tested against an elevation model.

In INC-0137 the range control antenna sits on a 12 m mast and the robot’s antenna 1.6 m up. The ground comes from the USGS 3DEP elevation grid at 10 m, and the berm, 6.5 m high, is added on top of it.

On the site’s elevation model, the berm stands above the line from the range control mast to UGV-05’s antenna for the whole gap. The robot kept driving, the collector spooled 31.2 MB, and the upload finished 3 min 48 s after range control heard it again. All 38,904 records were acknowledged.

The example’s report still calls the berm a hypothesis, one that rests on the elevation model.

Before anyone swaps a radio

1. Read telemetry_status on the companion’s link. If heartbeat_type_onboard_controller stayed true while heartbeat_type_gcs went false, the vehicle was fine as far as the companion.
2. Read the journal across the gap. Look for unit stops, kernel device messages and router errors.
3. Check the ground end. If the receiver logged other system IDs through the gap, it was working.
4. Test line of sight along the track. Use your own elevation data and the real antenna heights.
5. Log the radio if it speaks MAVLink. RADIO_STATUS reports rssi, remrssi, noise, remnoise, rxerrors and fixed, generated by the radio itself.² PX4 also flags a heartbeat from the MAV_COMP_ID_TELEMETRY_RADIO component in heartbeat_component_telemetry_radio.⁴

The mesh radios in these examples keep no log. The INC-0143 report names the cost: “Whether it rebooted, lost power or only lost the USB link during the 29.2 s is not observed.”