Jets, drones and ground robots

Built for the machines
you actually field.

Jet-powered UAS, multirotor drones and tracked or wheeled ground robots, rebuilt from the logs each part already writes.

Jet-powered UAS

Only the recorder saw the aircraft decide.

In a 5 g reversal 110 km from the ground site, the airframe hid its own belly antennas. The lost-link timer fired while the ground was blind, and the reconstruction shows it fired exactly on time.

Aircraft
Turbojet target-class UAS, 640 kg at the event, 4.2 m span
Test point
TP-7: 5.0 g level turn left, 180° reversal, FL150, M0.70, 110 km due east of the ground site
Lost-link logic
No valid uplink frame for 3.0 s: abort the test point, roll wings level, climb to FL160 and fly to the lost-link orbit
Sources read
The onboard IRIG 106 Chapter 10 recording (time, PCM, 1553, ARINC 429 and Ethernet channels), the ground telemetry recording and receiver log, the C2 ground station log, radar TSPI and the flight termination console log
Why the ground went blindIllustrative example. A turbojet target UAS rolls into a 5 g left turn at 17:42:03.000. The line of sight to the ground site rises above the belly plane, the uplink is lost at 03.580 and telemetry frame lock at 03.779. LOST LINK engages at 06.580, 3.000 s after the last valid uplink frame. Telemetry returns at 06.861 and the uplink at 07.140 during the roll-out. Flight termination is armed at 10.228 as a precaution, and the operator resumes the test card at 12.941.Vehicle time, UTC17:42:02.5 to 17:42:14.0030405060708091011121314EventsUplink lost, 3.560 sTM frame lock lost, 3.082 sA1A2A3A4A5A6A7A8A9Bank angleEGI, via 1553, left positiveLine of sightabove the belly plane, derivedReceived levelTM at the ground, C2 at the modemFTS signal strengthSSTO from PCM, 1 kHz78.5°, 5 g30°30° right, toward the lost-link orbit43.8°: TM frame lock lost32.8°: uplink below sensitivityWings level: the site is 2.7° below the belly plane-99.5 dBm: TM 1e-5 (Eb/N0 12 dB)-105.0 dBm: Decom lock and modem sensitivityTM, −76.2 dBm in the clearC2 uplink, −87.2 dBm in the clear1.66 V, 12 dB margin line3.63 V, ripple 3.42 to 3.84 V in the steep bankFCC modeTest pointLOST LINKResumedGround decomLockSearchLockFTSSafeARMUplink lostTM frame lock lostTMC2 uplinkIllustrative example. Shadow curve assumed: 1 dB per degree above 15°, capped at 32 dB.
Illustrative exampleSwipe sideways to see the whole chart. Every curve is computed from the aircraft’s attitude and the site geometry, and every event comes from the recorder, ground station and range logs. A10, when the range safety officer removes the arm tones at 17:42:38.532, is off the right edge.
A1A4A7A9Planned 180° reversalTM and C2 site, 110 km westN1 km grid
From above: 35.4° of the planned 180° were flown before LOST LINK rolled the wings level.
EventVehicle time, UTCSourceWhat it shows
A117:42:03.000Recorder, 1553 FCC statusFCC starts TP-7: roll into a 5 g left reversal at 60°/s
A217:42:03.580Recorder, C2 modem valid-frame counterLast valid uplink frame
A317:42:03.779Ground Chapter 10 PCM, LOCKSTDecommutator loses frame lock
A417:42:06.580Recorder, 1553 FCC statusLOST LINK, 3.000 s after the last valid frame; roll-out begins
A517:42:06.861Ground Chapter 10 PCM, LOCKSTFrame lock regained; the first frames already carry LOST LINK
A617:42:07.140Recorder, C2 modem valid-frame counterFirst valid uplink frame; the FCC stays in LOST LINK, as designed
A717:42:07.888Recorder, EGI attitude; radar TSPIWings level on 324.6°, then a 30° right turn toward the lost-link orbit
A817:42:10.228PCM FTS ARM discrete; FTS console logFlight termination armed as a precaution, 12.0 ms after the console command
A917:42:12.941Recorder, 1553 FCC status; C2 ground logOperator RESUME clears LOST LINK 37 ms after it was sent

Finding: the test card, not the aircraft

Had the turn continued, the same geometry returns the uplink after 4.55 s, against a 3.0 s timer. With the site on the inside of a 5 g turn at 110 km and a belly-only C2 antenna, the lost-link entry was certain. Fly the reversal with the site on the outside, add a top C2 antenna, or raise the timer for high-bank test points.

Here the clocks are not the problem

The recorder, the ground station and the range all run on IRIG-B, so their clocks agree to microseconds. What limits the order of events is sampling: 1 ms PCM frames, a 20 ms bus schedule and 50 ms radar samples. On the drone below, it is the other way around.

What the evidence shows

  • The uplink and telemetry losses start 46 ms and 138 ms after the line of sight climbs past 30° above the belly plane, and end as it falls back in the roll-out.
  • LOST LINK engaged 3.000 s after the last valid uplink frame, exactly as configured, and the aircraft flew the lost-link route.
  • The flight termination link never came near its margin: its lowest reading was 3.42 V against a 1.66 V line.

What it does not

  • That the airframe caused the shadowing. It is correlated, and the 15° knee and 1 dB per degree slope stay a hypothesis until an antenna pattern measurement confirms them.
  • The exact instant the flight computer set LOST LINK. Status words ride a 20 ms bus schedule, so the report places it to ±20 ms.
  • Why termination was armed. The console log records the command; the rule behind it lives in the range’s procedures.

Multirotor drones

The flight controller was fine. The companion computer wasn’t.

Telemetry stopped within a second of the kernel killing a service that had run out of memory. Nothing points at the radio: the MAVLink router stopped 19 ms after the service exited, and telemetry came back when the router did.

  1. 114:32:01.204perception_node memory above 6,144 MiB for 3 samplesProcesses
  2. 214:32:04.079Last obstacle_distance sample before a 7.5 s stallFlight log
  3. 314:32:04.118Kernel killed perception_node (out of memory)Journal
  4. 414:32:04.133perception.service main process exited, status 9/KILLJournal
  5. 514:32:04.152mavlink-router.service stoppedJournal
  6. 614:32:04.912Heartbeat due and not received. Nothing for 6.96 sReceive
  7. 714:32:09.214perception.service startedJournal
  8. 814:32:10.874Telemetry received againReceive
  • Example incident INC-0142
  • Companion journal (systemd)
  • Process memory, CPU and restarts
Scroll through it on the home page

Ground robots

Vibration, a USB radio and a 10 second timer.

In the obstacle belt, the mesh radio dropped off the companion computer’s USB bus 1.69 s after vibration crossed 24 m/s². PX4 held the vehicle 11.9 s after the last ground heartbeat reached it, not 10: its timer counts from the last status report that still carried the heartbeat.

INC-0143 UGV-02, T4 tracked UGV, at 15:41:14.293Z
GNSS receiveru-blox F9P, dual antennaVia the flight logMesh radioMANET, USB EthernetState not observed27Lidar32 beams, EthernetVia the journalBattery48 V Li-ion, 14SVia the flight logCompanion computerAGX Orin 64 GBJournal3Flight controllerPixhawk 6X, PX4 v1.16.0Flight log16Range controlGround stationNothing received4
  1. 115:41:00.912Vibration above 24 m/s² for 1 s in the obstacle belt
  2. 215:41:02.604Kernel: USB device 1-2.3 disconnected (mesh radio)
  3. 315:41:02.790mavlink-routerd cannot reach range control
  4. 415:41:03.713Heartbeat due and not received. Nothing for 31.0 s
  5. 515:41:05.286No ground station heartbeat on TELEM2
  6. 615:41:14.293GCS connection loss: switching to Hold
  7. 715:41:32.214Kernel: usb0 registered again
  8. 815:42:10.511Mission resumed from range control
  • Example incident INC-0143
  • Kernel and service events from the journal
  • PX4 Hold traced to COM_DL_LOSS_T
Why the 10 s timer fired at 11.9 s

What we need from your vehicle

The more sources, the tighter the ordering.

A flight log alone is enough to start. Each source you add pins more events to UTC and closes more of the gaps between them.

SourceWhat to sendNeedWhy
PX4 flight log.ulgRequiredThe completed ULog from the flight controller’s SD card.
Companion journaljournalctl -o json --since … --until …RecommendedThe systemd journal export covering the flight, with its boot ID.
Ground receive log.jsonlRecommendedWhat range control heard from the vehicle, and when.
Collector samples.jsonlOptionalProcess memory, CPU and restarts, if the collector was installed.

A pilot on your own data

Bring your hardest incident.

Send one failure you have already investigated. We rebuild it on your data, beside your current tools, and show where the evidence agrees with your conclusion and where it doesn’t.

Request a pilotDeployment and security
  1. 1
    Send one incidentA failure you have already investigated, with the flight log and whatever companion or ground evidence you kept.
  2. 2
    We reconstruct itBeside your current tools, on your data, with every claim traced to its source.
  3. 3
    Compare the answersWhere the evidence agrees with your conclusion, where it does not, and what it cannot decide.