Blog
Field notes on failure
Source-level write-ups for reliability and flight test engineers. PX4 internals, Linux on the companion computer, ground links, time and CUI handling, each with the primary sources it rests on.
FeaturedPX4 internals6 primary sourcesWhy PX4’s 10-second link-loss failsafe fires lateCOM_DL_LOSS_T does not start at the last heartbeat. It starts at the last status report that still carried it, which on a real vehicle adds one and a half to almost four seconds.Foxborne EngineeringSeptember 26, 20265 min readRead the post
Time and clocks6 primary sourcesBoot time, wall time, arrival time: one incident, three clocksA PX4 flight log counts from boot, the companion journal keeps wall time and the ground station logs arrival. Give each an explicit error bound, and you know which events you can put in order and which you cannot.Foxborne EngineeringSeptember 26, 20266 min readCompanion computers11 primary sourcesReading an OOM kill in a journalctl exportThe kernel’s out-of-memory report names the thread that asked, the process it killed and every page it counted. systemd then records the unit’s result, and two settings decide what happens next.Foxborne EngineeringSeptember 26, 20267 min readCompanion computers7 primary sourcesWhen BindsTo= takes your MAVLink router down with itA dependency in the unit graph can silence telemetry with no radio fault at all. Here is how to spot one in the journal, and how to prove it on the bench before anyone swaps a radio.Foxborne EngineeringSeptember 26, 20265 min readGround links5 primary sourcesA telemetry gap is not a radio failureA hole in the ground receive log proves only that nothing arrived. A stopped router, a radio that left the USB bus and a berm all look the same there, and other logs tell them apart.Foxborne EngineeringSeptember 26, 20266 min readCUI and handling8 primary sourcesCUI on page one: what 32 CFR 2002 and DoDI 5200.48 ask of an engineering reportThe government-wide rule asks for the same banner on every page and a designation indicator on the first. DoD practice prints plain CUI at top and bottom and moves the codes and a point of contact into a five-line block.Foxborne EngineeringSeptember 26, 20266 min readFlight test9 primary sourcesIRIG 106 for people who live in PX4 logsA range recorder stamps every packet with a 48-bit counter that ticks at 10 MHz, and it records absolute time as a channel of its own. If you can put a ULog on UTC, you already know the method.Foxborne EngineeringSeptember 26, 20267 min readFlight test6 primary sourcesAt jet speed, clock error is distanceAt 170 m/s, a ±40 ms clock bound is ±6.8 m of track; on a quad it is ±0.48 m. Draw it along the track, and you can see when a log cannot say which side of a line an event fell on.Foxborne EngineeringSeptember 26, 20266 min readPX4 internals6 primary sourcesWhy PX4’s 10-second link-loss failsafe fires lateCOM_DL_LOSS_T does not start at the last heartbeat. It starts at the last status report that still carried it, which on a real vehicle adds one and a half to almost four seconds.Foxborne EngineeringSeptember 26, 20265 min read
A pilot on your own data
Bring your hardest incident.
Send one failure you have already investigated. We rebuild it on your data, beside your current tools, and show where the evidence agrees with your conclusion and where it doesn’t.
- 1Send one incidentA failure you have already investigated, with the flight log and whatever companion or ground evidence you kept.
- 2We reconstruct itBeside your current tools, on your data, with every claim traced to its source.
- 3Compare the answersWhere the evidence agrees with your conclusion, where it does not, and what it cannot decide.