The product

From four log files
to one verified sequence.

Import, align, reconstruct and hand off. Every step is recorded, and every result can be rerun to the same answer.

  • Deterministic rules
  • Order unknown inside the error
  • Seven export formats
  • No invented confidence scores

01 Import

Drop in four files, or point at your bucket.

Foxborne recognizes each file by its content, hashes it on arrival and never writes to it again. The import job lists every step it ran, so nothing happens out of sight.

INC-01424 originals
14_13_19.ulgPX4 ULog
f60310be1687…f69f
uas04-orin-journal.jsonjournalctl JSON
a0741c955f33…cdf4
uas04-orin-samples.jsonlCollector JSONL
c4d46f086f9c…8830
rc1-receive.jsonlReceiver JSONL
a96027361d50…8905
  1. RecognizeEach file identified by its content: the ULog header, journal fields and the collector and receiver schemas.
  2. HashSHA-256 of every original, stored write-once before anything reads it.
  3. Parse the flight log11 of 13 expected topics present. Not logged: distance_sensor and vehicle_imu_status.
  4. Parse the journalRealtime and monotonic timestamps kept, with the boot ID that ties the collector samples to it.
  5. AlignFlight log ±40 ms, journal ±6 ms, ground receive ±15 ms plus latency. Aligned ±46 ms.
  6. Run the rules12 deterministic rules, each finding stamped with its rule version and threshold.
  • PX4 ULog
  • journalctl JSON
  • Collector JSONL
  • Receiver JSONL
  • IRIG 106 Chapter 10
  • Range records

02 Align

Every source on UTC, with its error drawn in.

GPS time anchors the flight log, chrony vouches for the companion clock and range control stamps every arrival. Each source keeps its own error bound, and no line of the report claims an order the bounds cannot support.

Flight log

Counts microseconds from boot. GPS time in sensor_gps pins it to UTC.

the moment it happened
boot + 1,132.832 s±40 ms
Companion journal

Keeps wall time, which holds only while chrony reports a synchronized source.

14:32:04.118±6 ms
Ground receive log

Records when a packet arrived, not when it left. The moment it describes is earlier.

−200 ms−100 ms0+100 ms+200 mssent up to 412 ms before it arrived
14:32:04.214±15 ms + latency
Above a 2 s bound
When two sources cannot be aligned within two seconds, Foxborne keeps them on separate clocks and makes no before or after claim between them.
No claim
  • Error bound per source
  • 2 s alignment limit

03 Reconstruct

Rules catch what changed. You decide what it means.

12 deterministic rules run on every import. Each finding records the rule, its version and the threshold that fired, so the same files always give the same findings.

RuleDetectsThresholdIn the report
Service exitprocess_exit v1.2An allowlisted systemd unit leaves the active state.6 units on the allowlistObserved: process exited
Service restartservice_restart v1.0systemd schedules a restart of an allowlisted unit.Any restartObserved: service restarted
Memory pressurememory_pressure v1.1A watched process stays above its memory threshold.RSS at or above 6,144 MiB for 3 samplesObserved: memory rose above threshold
Telemetry gaptelemetry_gap v1.3No HEARTBEAT arrives at the ground receiver beyond the expected interval.More than 3 s without a HEARTBEATObserved: telemetry receive gap, not necessarily radio failure
Device disconnectdevice_disconnect v1.0The kernel reports that a USB device on the allowlist disconnected.Any disconnect on an allowlisted portObserved: device disconnected
Vibrationvibration_high v1.0accel_vibration_metric in vehicle_imu_status stays above the platform threshold.Above 24 m/s² for 1 s (T4), 12 m/s² (Q4)Observed: vibration above threshold
Failsafefailsafe_entered v1.0vehicle_status reports a failsafe, with the cause from failsafe_flags.Any failsafeObserved: failsafe entered, with its cause
Topic stalltopic_stall v1.0A logged uORB topic stops updating at its expected rate.No sample for 5 times the median periodObserved: topic stopped updating
GNSS jamminggnss_jamming v1.0sensor_gps reports jamming_state 3 (detected) from the receiver.jamming_state 3 for 1 sObserved: the receiver reported jamming
GPS quality dropgps_quality_drop v1.0Horizontal accuracy or satellite count crosses the vehicle threshold.eph above 2.5 m or fewer than 8 satellites for 2 sObserved: GPS quality degraded
Logger dropoutlog_dropout v1.0The flight log records a dropout marker.Dropout of 50 ms or longerObserved: flight log has a dropout
Clock uncertainclock_uncertain v1.0A source has no trustworthy anchor, or its estimated error exceeds the ordering bound.Anchor missing or error above 2 sWarning: ordering between sources is uncertain

04 Map and vehicle

Where it happened, and which part saw it.

Maps are drawn from the elevation data you load, with the MGRS grid, the track and every stretch the ground heard nothing. No tile service is ever called, so an air-gapped network shows the same map.

75768687Range controlReceive log, sysid 41.20 kmUAS-04 at 14:32:04.118Z13R FP 7639 8764The kernel kills perception_nodeNothing received for 6.96 s
Example incident INC-01421 km MGRS grid, contours every 2 m. The amber stretch is where range control heard nothing.
INC-0143 UGV-02, T4 tracked UGV, at 15:41:14.293Z
GNSS receiveru-blox F9P, dual antennaVia the flight logMesh radioMANET, USB EthernetState not observed27Lidar32 beams, EthernetVia the journalBattery48 V Li-ion, 14SVia the flight logCompanion computerAGX Orin 64 GBJournal3Flight controllerPixhawk 6X, PX4 v1.16.0Flight log16Range controlGround stationNothing received4
  1. 115:41:00.912Vibration above 24 m/s² for 1 s in the obstacle belt
  2. 215:41:02.604Kernel: USB device 1-2.3 disconnected (mesh radio)
  3. 315:41:02.790mavlink-routerd cannot reach range control
  4. 415:41:03.713Heartbeat due and not received. Nothing for 31.0 s
  5. 515:41:05.286No ground station heartbeat on TELEM2
  6. 615:41:14.293GCS connection loss: switching to Hold
  7. 715:41:32.214Kernel: usb0 registered again
  8. 815:42:10.511Mission resumed from range control

05 Evidence levels

Measured, inferred or unknown.

Every line carries one of four levels, and two events inside each other’s error bars stay unordered. Move along the timeline: the detail below it is 140 ms of the example, drawn 100 times closer.

Observed

A value, event or log line captured directly.

perception.service exited at 14:32:04.133.

Correlated

Events that line up inside a known error bound.

The next heartbeat was due 0.76 s after the router stopped.

Hypothesis

An explanation that still needs a check.

The router unit may be bound to the perception unit.

Unknown

Sources are missing, partial, or disagree on time.

Nothing supports an order between the stall and the kill.

Example incident timelineFour sources on one UTC axis. Memory rises above its threshold at 14:32:01.204, the kernel kills perception_node at 14:32:04.118, telemetry stops, and the service restarts at 14:32:09.214. A detail view shows the last obstacle_distance sample and the kernel kill 39 ms apart, inside a 46 ms combined clock error, so their order is unknown.INC-0142Telemetry gap after perception.service was killedAligned ±46 msUTC, 24 Sep 2026FindingsRule resultsFlight controller14_13_19.ulg±40 msCompanion journaluas04-orin±6 msProcess samplesperception_node±6 msGround receivesysid 4, 1 Hz±15 ms14:32:0014:32:0214:32:0614:32:0814:32:1014:32:12MissionThreshold 6,144 MiB6,800 MiB5,800 MiBrestarted at 412 MiB, below this scaleMemory above thresholdperception.service killed, and 2 moreTelemetry gap 6.96 sService restartedTelemetry back14:32:04.118Every event is drawn as an interval as wide as its clock error.No dataOrder unknownFindingDetail14:32:04.030 to 14:32:04.170, drawn 100 times closerFlight logJournal04.05004.10004.150obstacle_distance, last sample, ±40 msKernel OOM kill and systemd, ±6 msOrder unknown39 ms apart, inside ±46 ms
  1. 14:32:01.204±6 ms
    perception_node memory passed 6,144 MiB
    Observed
  2. 14:32:04.079±40 ms
    Last obstacle_distance sample before a 7.5 s stall
    Order unknown
  3. 14:32:04.118±6 ms
    Kernel killed perception_node, out of memory
    Order unknown
  4. 14:32:04.152±6 ms
    mavlink-router.service stopped
    Observed
  5. 14:32:04.912±15 ms
    Heartbeat due and not received. Nothing for 6.96 s
    Correlated
  6. 14:32:09.214±6 ms
    perception.service restarted
    Observed
  7. 14:32:10.874±15 ms
    Telemetry received again
    Observed

06 Hand off

A report another engineer can check.

Findings, sequence, evidence, limitations and open questions, on pages that carry your marking. Every excerpt names its file, its line or byte offset and the SHA-256 of the file.

  • CUI banner and designation indicator
  • File, line and byte offset
  • Open questions travel with the report
CUI
Controlled by
Harrow Field Robotics
Controlled by
Reliability Engineering
CUI category
EXPT
Dissemination
FEDCON
POC
[email protected]
Incident report INC-0142

Telemetry gap after perception.service was killed

1Finding
Hypothesis

Telemetry stopped reaching range control within one second of the kernel killing perception_node. The router unit may be bound to the perception unit.

Written by the investigating engineer. Everything below is extracted from the sources.
2Sequence
Time, UTCEventLevelRef
14:32:04.079±40 msLast obstacle_distance sample
Order against E3 is unknown: 39 ms apart, inside ±46 ms.
Order unknownE2
14:32:04.118±6 msKernel killed perception_nodeOrder unknownE3
14:32:04.152±6 msmavlink-router.service stoppedObservedE5
14:32:04.912±15 msHeartbeat due and not received
0.76 s after the router stopped.
CorrelatedE6
3Evidence
E3
Kernel killed perception_node
uas04-orin-journal.json, line 18,314, sha256:a0741c955f33…cdf4
Out of memory: Killed process 2213 (perception_node) total-vm:11873248kB, anon-rss:6823516kB
  1. 1
    The engineer’s conclusion, labeledA finding says whether it is an observation, a hypothesis or unknown, and who wrote it. Everything below it is extracted from the sources.
  2. 2
    Uncertainty stays visibleTwo events inside each other’s clock error are marked order unknown in the report, exactly as on screen.
  3. 3
    Every line points at bytesThe file, the line or byte offset, and the SHA-256 of the file. Anyone holding the original can check each line.
  4. 4
    Your marking on every pageThe banner marking at the top and bottom of every page, and the CUI designation indicator on the first, set once by your administrator.
PDF case file with CUI markingsThe report, with CUI in the banner and footer of every page and the designation indicator on the first.
BagIt evidence bundleAn RFC 8493 bag with SHA-256 and SHA-512 manifests and the case metadata in bag-info.txt.
Original files with SHA-256 manifestByte-identical originals and a sha256sum-style manifest, checked again on export.
MCAPThe aligned timeline for Foxglove and ROS 2 tools, with the case ID and source SHA-256s in its metadata.
Apache ParquetOne table per signal family, UTC timestamps in microseconds, for fleet analytics.
CSV and JSONLCSV for spreadsheets and scripts, JSON Lines for SIEM and stream loaders.
MATLAB .mat v7.3HDF5-based, for flight-test engineers’ MATLAB and Simulink tools.

07 AI models

AI where a rule cannot reach. Never where it would decide.

Six jobs that a fixed rule or a hand-written parser does badly. Every proposal is checked by code before anything changes, and none of them writes evidence, a finding or an order between events.

  • Runs inside your boundary
  • Ships in the signed bundle
  • Nothing sent to a model provider
  • Proposes, never concludes
WhereWhat the model doesWhat checks it
Read a log nobody wrote a parser forReads the first lines of an unfamiliar text log, such as a radio’s status export, and proposes the time column, its clock and what each column holds.The parser runs over every line before an admin saves it. Lines it cannot read are listed, never guessed.
Find one moment on two clocksFinds events on an unsynchronized companion clock that have a counterpart in the flight log, so the clock can be aligned by hand.The delay between them is learned from aligned runs, and a pair seen too rarely cannot be used. The offset and its bound are arithmetic.
Check a finding against the recordsLists every time, time difference, value and parameter an engineer’s finding states, and every word that claims a cause.Each claim is matched to a record and to the clock bounds. The text is marked, never changed.
Turn an investigation into a ruleReads incident findings for two rules that fire in order on one vehicle, such as a disconnect seconds after a vibration spike.The runs where both fired are counted, and the rule runs nowhere until an admin creates it.
Explain a raw recordReads a flight log message or a journal entry field by field, and says what the record cannot tell you.The record stays on screen as the evidence. The reading never enters a report.
Search by meaningFinds evidence that says the same thing in other words: a search for radio finds USB disconnects and heartbeat losses.Every result is an event with its ID and its original record.

08 Boundaries

Read-only by design.

No command channel, no open port on the vehicle, no data sent to us. It is how Foxborne is built, not a setting someone can change.

No command channel

The collector never sends MAVLink to the flight controller. Foxborne cannot fly, command or reconfigure a vehicle.

No open port on the vehicle

Collectors connect outbound over mutual TLS and spool on board while the link is down.

Engineers conclude

Rules report what they observed. The engineer writes the conclusion, and the report keeps the two apart.

No invented confidence scores

A figure like 91% needs a calibrated model and labeled incidents. Reviewers get the evidence itself, which is what they sign off on.

Write-once originals

Files are hashed on arrival and never rewritten. Reviewers annotate beside the evidence, never on top of it.

Nothing sent to us

A self-hosted deployment has no telemetry, no crash reporting and no license call home.

A pilot on your own data

Bring your hardest incident.

Send one failure you have already investigated. We rebuild it on your data, beside your current tools, and show where the evidence agrees with your conclusion and where it doesn’t.

  1. 1
    Send one incidentA failure you have already investigated, with the flight log and whatever companion or ground evidence you kept.
  2. 2
    We reconstruct itBeside your current tools, on your data, with every claim traced to its source.
  3. 3
    Compare the answersWhere the evidence agrees with your conclusion, where it does not, and what it cannot decide.