Incident reconstruction for field robotics
Know what failed first.
Prove it line by line.
Rebuild field failures from the logs your robots already keep. One timeline, one map, and a report where every finding cites the line that proves it.
- Time
- 14:32:04.118Z
- Position
- 13R FP 7639 8764
- Altitude, speed
- 59.5 m AGL, 5.4 m/s
- Ground link
- Nothing received, 1.20 km
- Runs in your AWS GovCloud account
- Air-gapped install
- PX4 ULog, systemd journal, MAVLink
- SHA-256 on every original
The example incident
Four logs on one clock. Scrub it, and the map follows.
INC-0142: telemetry stopped within a second of the kernel killing a service that had run out of memory. Every event is drawn as wide as its clock error, so you can see which ones the evidence can put in order.
- 14:32:01.204±6 msperception_node memory passed 6,144 MiBObserved
- 14:32:04.079±40 msLast obstacle_distance sample before a 7.5 s stallOrder unknown
- 14:32:04.118±6 msKernel killed perception_node, out of memoryOrder unknown
- 14:32:04.152±6 msmavlink-router.service stoppedObserved
- 14:32:04.912±15 msHeartbeat due and not received. Nothing for 6.96 s
- 14:32:09.214±6 msperception.service restartedObserved
- 14:32:10.874±15 msTelemetry received againObserved
Move along the timeline and the vehicle moves on the map above. The flight log’s last obstacle sample and the kill are 39 ms apart, inside a 46 ms combined clock error, so Foxborne does not put them in order.
Step through it
Seven moments. Each one cites the line that recorded it.
Scroll, and the incident clock moves with you. The map, the merged log from all four sources and the time ruler follow it, one finding at a time.
14:31:59.500ZBefore anything breaks
UAS-04 is flying Route Iron.
59.5 m above the ground and 1.22 km from range control, which hears it once a second. On the companion computer, perception_node holds 6,131 MiB.
14:32:01.204ZProcess, ±6 ms
The first sign, 2.9 s before anything breaks.
On the companion computer, perception_node passes 6,144 MiB and keeps climbing, to 6,655 MiB by 14:32:03.204Z. The memory pressure rule fires. Range control is still hearing the drone once a second.
Observed14:32:04.118ZJournal, ±6 ms
The kernel kills perception_node.
Out of memory, in the companion journal at 14:32:04.118Z. The flight log’s last obstacle sample is 39 ms earlier, inside the 46 ms the two clocks can disagree by. Foxborne marks the order unknown instead of guessing.
Order unknown14:32:04.152ZJournal, ±6 ms
The telemetry path goes down with it.
34 ms later, systemd has stopped mavlink-router, the process that bridges the flight controller to the mesh radio. Why it stopped stays a hypothesis until someone reads the unit files: it may be bound to perception.service.
Why: hypothesis14:32:04.912ZRange control, ±15 ms
Range control hears nothing.
The next heartbeat was due at 14:32:04.912Z and never came. The receive log stays empty for 6.96 s. The radio keeps no log of its own, so Foxborne claims only what the evidence shows: nothing arrived.
Observed14:32:09.214ZJournal, ±6 ms
systemd brings both back.
The router starts at 14:32:09.203Z and perception.service 11 ms later, 5.1 s after the kill. The router reopens the mesh radio link at 14:32:10.718Z.
Observed14:32:10.874ZRange control, ±15 ms
Telemetry is back after 6.96 s.
Range control hears the drone again at 14:32:10.874Z. On board, PX4 flagged the missing ground heartbeat on TELEM2 at 14:32:06.510Z and logged it restored at 14:32:12.944Z.
ObservedThe finding
One sequence. Every line cited.
- Observedperception.service exited at 14:32:04.133.
- The next heartbeat was due 0.76 s after the router stopped.
- HypothesisThe router unit may be bound to the perception unit.
- UnknownNothing supports an order between the stall and the kill.
- Range controlHEARTBEAT from system 4
- Processperception_node RSS 6,131 MiB
Platforms
Drones, jets and ground robots. One method.
The faster the vehicle, the more a clock error costs. In the example, the flight log and the journal can disagree by 46 ms. Foxborne carries that bound into every line, so an order the clocks cannot prove never reads as fact.
The hand-off
A report your safety review board can check line by line.
Findings, sequence and evidence on marked pages, with the file, line and SHA-256 behind every statement. What the engineer concluded stays apart from what the sources show.
- CUI banner and designation indicator
- File, line and byte offset
- Open questions travel with the report
- Controlled by
- Harrow Field Robotics
- Controlled by
- Reliability Engineering
- CUI category
- EXPT
- Dissemination
- FEDCON
- POC
- [email protected]
Telemetry gap after perception.service was killed
Telemetry stopped reaching range control within one second of the kernel killing perception_node. The router unit may be bound to the perception unit.
| Time, UTC | Event | Level | Ref |
|---|---|---|---|
| 14:32:04.079±40 ms | Last obstacle_distance sample Order against E3 is unknown: 39 ms apart, inside ±46 ms. | Order unknown | E2 |
| 14:32:04.118±6 ms | Kernel killed perception_node | Order unknown | E3 |
| 14:32:04.152±6 ms | mavlink-router.service stopped | Observed | E5 |
| 14:32:04.912±15 ms | Heartbeat due and not received 0.76 s after the router stopped. | E6 |
- 1The engineer’s conclusion, labeledA finding says whether it is an observation, a hypothesis or unknown, and who wrote it. Everything below it is extracted from the sources.
- 2Uncertainty stays visibleTwo events inside each other’s clock error are marked order unknown in the report, exactly as on screen.
- 3Every line points at bytesThe file, the line or byte offset, and the SHA-256 of the file. Anyone holding the original can check each line.
- 4Your marking on every pageThe banner marking at the top and bottom of every page, and the CUI designation indicator on the first, set once by your administrator.
AI models
Models that check the work. Engineers still conclude.
Foxborne uses AI models where a rule cannot reach, from reading an unfamiliar log to checking a finding’s figures against the records. They run inside your boundary, and every proposal is checked by code before anything changes.
Reads the first lines of an unfamiliar text log, such as a radio’s status export, and proposes the time column, its clock and what each column holds.
Finds events on an unsynchronized companion clock that have a counterpart in the flight log, so the clock can be aligned by hand.
Lists every time, time difference, value and parameter an engineer’s finding states, and every word that claims a cause.
Reads incident findings for two rules that fire in order on one vehicle, such as a disconnect seconds after a vibration spike.
Reads a flight log message or a journal entry field by field, and says what the record cannot tell you.
Finds evidence that says the same thing in other words: a search for radio finds USB disconnects and heartbeat losses.
Deployment
Runs inside your boundary. Sends nothing out.
Install it in your AWS GovCloud account, your data center or an air-gapped enclave. It reads the buckets and shares you already run, keeps originals under your keys, and never phones home.
Serves the console on your network only.
Parse, align and run rules on your hardware.
Events, notes, reports and the audit log.
Originals, written once, under your key.
Keys stay there. Revoke one and the data is unreadable.
SAML or OIDC sign-in. Audit events out.
- AWS GovCloud
- Azure Government
- Air-gapped
- Customer-managed keys
- SAML or OIDC
The collector never sends MAVLink to the flight controller. Foxborne cannot fly, command or reconfigure a vehicle.
Collectors connect outbound over mutual TLS and spool on board while the link is down.
Rules report what they observed. The engineer writes the conclusion, and the report keeps the two apart.
A figure like 91% needs a calibrated model and labeled incidents. Reviewers get the evidence itself, which is what they sign off on.
Files are hashed on arrival and never rewritten. Reviewers annotate beside the evidence, never on top of it.
A self-hosted deployment has no telemetry, no crash reporting and no license call home.
Field notes
Written from the source, for engineers.
PX4 internals, Linux on the companion computer, ground links and time, each post built on the primary sources it cites.
A pilot on your own data
Bring your hardest incident.
Send one failure you have already investigated. We rebuild it on your data, beside your current tools, and show where the evidence agrees with your conclusion and where it doesn’t.
- 1Send one incidentA failure you have already investigated, with the flight log and whatever companion or ground evidence you kept.
- 2We reconstruct itBeside your current tools, on your data, with every claim traced to its source.
- 3Compare the answersWhere the evidence agrees with your conclusion, where it does not, and what it cannot decide.