# Foxborne: full index Foxborne is incident reconstruction for field robotics (drones and ground robots running PX4 with a Linux companion computer, and jet-powered UAS with IRIG 106 Chapter 10 flight test recordings), sold and supported by Dude Lemon LLC. Site: https://foxborne.com. ## How it works 1. **Import.** Drop in a .ulg, a journalctl export and collector samples, or point Foxborne at the bucket where they already land. Every original is hashed the moment it arrives. 2. **Align.** GPS time anchors the flight log and chrony vouches for the companion clock, so every source lands on UTC with a stated error. Anything unanchored stays visibly separate. 3. **Reconstruct.** Rules catch the memory climb, the service exit and the telemetry gap. Each finding names the rule version and threshold that fired, so a rerun next month gives the same answer. 4. **Hand off.** Export a report with exact excerpts, the SHA-256 of every source file, the limits of the evidence and the open questions, under your marking. ## Formats - **PX4 ULog.** Completed flight logs: vehicle status, position, battery, GPS, telemetry, logged messages and dropouts. - **journalctl JSON.** systemd journal exports from the companion computer, with realtime and monotonic timestamps and the boot ID. - **Collector JSONL.** Process samples from the Foxborne collector: CPU, resident memory and restart count, with clock status on every sample. - **Receiver JSONL.** Ground station receive records: which MAVLink message arrived, from which system, and when it arrived. - **IRIG 106 Chapter 10.** Flight test recordings: time packets, PCM, MIL-STD-1553 and ARINC 429 channels, decoded with the recording’s own TMATS setup record. - **Range records.** Radar TSPI, telemetry receiver and flight termination console logs on range time, mapped once per range. ## Clocks - **Flight log.** Counts microseconds from boot. GPS time in sensor_gps pins it to UTC. Typical bound ±40 ms. - **Companion journal.** Keeps wall time, which holds only while chrony reports a synchronized source. Typical bound ±6 ms. - **Ground receive log.** Records when a packet arrived, not when it left. The moment it describes is earlier. Typical bound ±15 ms + latency. - Above a 2 s bound, Foxborne keeps sources on separate clocks and makes no before or after claim between them. ## Evidence ladder - **Observed.** A value, event or log line captured directly. Example: “perception.service exited at 14:32:04.133.” - **Correlated.** Events that line up inside a known error bound. Example: “The next heartbeat was due 0.76 s after the router stopped.” - **Hypothesis.** An explanation that still needs a check. Example: “The router unit may be bound to the perception unit.” - **Unknown.** Sources are missing, partial, or disagree on time. Example: “Nothing supports an order between the stall and the kill.” - No confidence percentages are printed. ## Deployment - **Your cloud account.** Deploys into your AWS, AWS GovCloud or Azure Government account. Originals stay in your buckets, encrypted with your keys. - **Your data center.** Runs on your own Kubernetes or a pair of Linux hosts, next to the log servers you already have. - **Air-gapped.** No network path out at all. Updates arrive as signed bundles that your administrator carries in and verifies. - **Original files** live in your object storage. Hashed with SHA-256 on arrival, stored write-once, encrypted with your key. - **Normalized events and notes** live in postgresql in your network. Stable event IDs: the same file and parser version always give the same IDs. - **Reports and case files** live in your object storage. Every excerpt quoted exactly, with the SHA-256 of its file. Your marking on every page. - **Basemaps and elevation** live in your object storage. Maps are drawn from elevation data you load. Nothing calls a tile service, so an air-gapped network shows the same map. - **Audit log** live in append-only table, streamed to your siem. Each entry carries the hash of the one before it, so gaps and edits are detectable. ### Identity - Single sign-on over SAML 2.0 or OIDC with your identity provider. - SCIM provisioning, so removing someone upstream removes them here. - Five fixed roles: admin, investigator, reviewer, viewer and auditor. - A break-glass local account that requires a hardware security key. ### Audit - Every import, view, edit, export and admin change is recorded. - Entries are hash-chained and can be verified at any time. - Streams to your SIEM over syslog with TLS. ### Markings and export control - A system-high marking banner in the standard classification colors, set by your administrator. - The CUI designation indicator on the first page of every report and case file. - Evidence marked export-controlled opens only for users your identity provider asserts are US persons. ### Supply chain - Releases are signed bundles with a software bill of materials. - The signature is checked before anything installs. - Licenses are checked offline, with no call home. ### Vehicle collector - Outbound only, over mutual TLS, with a certificate per vehicle. - Reads an allowlist of journal units and processes, and nothing else. - Redacts configured patterns on the vehicle, before upload. - Never collects environment variables or process arguments. ## Platforms - **Jet-powered UAS** (Illustrative example). Lost link in a 5 g reversal, 110 km from the ground site. Sources: Onboard IRIG 106 Chapter 10 recording; Ground telemetry recorder and receiver; Radar TSPI and the FTS console log. - **Multirotor drones** (Example incident INC-0142). Telemetry stopped within a second of the kernel killing a service. Sources: PX4 ULog from the flight controller; systemd journal from the companion; Range control receive records. - **Ground robots** (Example incident INC-0143). A radio fell off the USB bus in the obstacle belt, and PX4 held. Sources: PX4 ULog with vehicle_imu_status; Kernel and service events from the journal; Range control receive records. ## Reads from - **Amazon S3.** Including AWS GovCloud, through a read-only IAM role. - **S3-compatible storage.** MinIO, Ceph or any S3 API inside your network. - **Azure Blob Storage.** Including Azure Government, through a managed identity. - **Google Cloud Storage.** Through a service account with the viewer role. - **SFTP.** Ground stations and log servers, polled on a schedule. - **SMB or NFS.** Network shares mounted read-only on the worker. - **Companion collector.** A read-only service on each companion computer. - **MAVLink receiver.** Records every receipt on the ground with its own clock. - **Ingest API.** Test rigs and pipelines push signed bundles. - **Manual upload.** Drop files into the console. Hashed in the browser first. ## Alerts - **R1 Vibration above threshold.** accel_vibration_metric above 24 m/s² for 1.0 s while armed, on the tracked ground robot profile. Warning, High if the IMU clips. - **R2 Telemetry gap.** More than 3.0 s between heartbeats from the vehicle’s system ID while armed. Medium, High once the gap reaches COM_DL_LOSS_T. - **R3 Failsafe entered.** vehicle_status.failsafe goes true while armed, with the cause from failsafe_flags. High: opens a case and a ticket. - **R4 Memory pressure.** A process at or above 6,144 MiB RSS for 3 samples, critical on a kernel OOM kill. Warning, then Critical. - **R5 Device disconnect.** A registered USB device or CAN interface disconnects or goes bus-off while armed. High for flight-critical devices. - **R6 GNSS jamming.** jamming_state 3 for 2 s, or jamming_indicator at 80 or above for 5 s. High, routed to range safety. ## Automations - **A1 Sortie closeout triage.** When every source on the test card has arrived, Foxborne verifies the manifest, aligns the clocks and runs the rules. A high finding opens a case and a ticket. - **A2 Failsafe evidence pack.** An unplanned failsafe freezes 120 s either side across every source and drafts a marked case file. Nothing leaves until a named approver signs off. - **A3 Repeat hardware fault.** The same device failing on 3 of a vehicle’s last 10 sorties opens a maintenance ticket and flags the vehicle for inspection. - **A4 Case closure and retention.** An approved case becomes a BagIt bundle on write-once storage, with its manifest hash recorded in the ticket and the audit log. ## Destinations - **L0 Metadata.** Rule, severity, vehicle alias, UTC time and a link back into Foxborne. The default for anything outside the enclave. - **L1 Summary.** The finding with its values. Treated as CUI, and sent only to destinations your program has authorized for it. - **L2 Evidence.** Files and excerpts. They move only through Export, after a named approver signs off. - **Notify: Email.** SMTP relay, STARTTLS on 587 or TLS on 465. Works air-gapped. - **Notify: Mattermost.** Incoming webhook or bot token. Works air-gapped. - **Notify: Signed webhook.** HTTPS JSON, HMAC-SHA256, optional mutual TLS. Works air-gapped. - **Notify: Alertmanager.** Alertmanager API v2. Works air-gapped. - **Notify: Microsoft Teams, GCC High and DoD.** Teams bot or Microsoft Graph. Needs a network path to a government cloud. - **Notify: GovSlack.** Webhook or Web API. Needs a network path to a government cloud. - **Notify: PagerDuty.** Events API v2, metadata only. Needs a network path to a government cloud. - **Notify: Amazon SNS in GovCloud.** SNS Publish with an IAM role. Needs a network path to a government cloud. - **Ticketing: Jira Data Center.** REST, personal access token. Works air-gapped. - **Ticketing: GitLab self-managed.** Issues API or alert endpoint. Works air-gapped. - **Ticketing: Jira in Atlassian Government Cloud.** REST, OAuth 2.0. Needs a network path to a government cloud. - **Ticketing: ServiceNow.** Table API or Event Management. Needs a network path to a government cloud. - **Forward logs: Syslog over TLS.** RFC 5424 on RFC 5425, mutual X.509. Works air-gapped. - **Forward logs: Splunk HEC.** HTTPS JSON with acknowledgement. Works air-gapped. - **Forward logs: Elasticsearch or OpenSearch.** Bulk API, API key or mutual TLS. Works air-gapped. - **Forward logs: Kafka.** Mutual TLS or SASL, with ACLs. Works air-gapped. - **Forward logs: OpenTelemetry collector.** OTLP over HTTP. Works air-gapped. - **Forward logs: Microsoft Sentinel.** Logs Ingestion API. Needs a network path to a government cloud. - **Forward logs: Amazon SQS or EventBridge.** IAM role. Needs a network path to a government cloud. - Every destination starts disabled. An administrator enables it with a host allowlist, credentials and a data class. - Markings travel with the content. Summaries and evidence carry your program’s banner. - No program or vehicle names in cloud resource names, tags or subjects. - Every delivery lands in a ledger with its destination, data class, payload SHA-256 and response. - At-least-once delivery with idempotency keys, backoff, a dead-letter queue and replay. ## Exports - **PDF case file with CUI markings.** The report, with CUI in the banner and footer of every page and the designation indicator on the first. - **BagIt evidence bundle.** An RFC 8493 bag with SHA-256 and SHA-512 manifests and the case metadata in bag-info.txt. - **Original files with SHA-256 manifest.** Byte-identical originals and a sha256sum-style manifest, checked again on export. - **MCAP.** The aligned timeline for Foxglove and ROS 2 tools, with the case ID and source SHA-256s in its metadata. - **Apache Parquet.** One table per signal family, UTC timestamps in microseconds, for fleet analytics. - **CSV and JSONL.** CSV for spreadsheets and scripts, JSON Lines for SIEM and stream loaders. - **MATLAB .mat v7.3.** HDF5-based, for flight-test engineers’ MATLAB and Simulink tools. ## AI models AI models run on an inference node inside the deployment and ship in the signed release bundle. Nothing goes to a model provider. They never write evidence, a finding or an order between events; each proposal is checked by code before anything changes. - **Read a log nobody wrote a parser for.** Reads the first lines of an unfamiliar text log, such as a radio’s status export, and proposes the time column, its clock and what each column holds. The parser runs over every line before an admin saves it. Lines it cannot read are listed, never guessed. Docs: https://docs.foxborne.com/collect/import-files - **Find one moment on two clocks.** Finds events on an unsynchronized companion clock that have a counterpart in the flight log, so the clock can be aligned by hand. The delay between them is learned from aligned runs, and a pair seen too rarely cannot be used. The offset and its bound are arithmetic. Docs: https://docs.foxborne.com/investigate/align-a-clock - **Check a finding against the records.** Lists every time, time difference, value and parameter an engineer’s finding states, and every word that claims a cause. Each claim is matched to a record and to the clock bounds. The text is marked, never changed. Docs: https://docs.foxborne.com/investigate/write-a-report - **Turn an investigation into a rule.** Reads incident findings for two rules that fire in order on one vehicle, such as a disconnect seconds after a vibration spike. The runs where both fired are counted, and the rule runs nowhere until an admin creates it. Docs: https://docs.foxborne.com/rules/configure - **Explain a raw record.** Reads a flight log message or a journal entry field by field, and says what the record cannot tell you. The record stays on screen as the evidence. The reading never enters a report. Docs: https://docs.foxborne.com/investigate/event - **Search by meaning.** Finds evidence that says the same thing in other words: a search for radio finds USB disconnects and heartbeat losses. Every result is an event with its ID and its original record. Docs: https://docs.foxborne.com/get-started/console-tour ## Read-only by design - **No command channel.** The collector never sends MAVLink to the flight controller. Foxborne cannot fly, command or reconfigure a vehicle. - **No open port on the vehicle.** Collectors connect outbound over mutual TLS and spool on board while the link is down. - **Engineers conclude.** Rules report what they observed. The engineer writes the conclusion, and the report keeps the two apart. - **No invented confidence scores.** A figure like 91% needs a calibrated model and labeled incidents. Reviewers get the evidence itself, which is what they sign off on. - **Write-once originals.** Files are hashed on arrival and never rewritten. Reviewers annotate beside the evidence, never on top of it. - **Nothing sent to us.** A self-hosted deployment has no telemetry, no crash reporting and no license call home. ## Questions a security review asks **Does any data leave our network?** Not in a self-hosted or air-gapped deployment. Foxborne sends no telemetry, crash reports or license checks. The only outbound traffic is what you configure, such as audit events to your SIEM. **Can Foxborne staff access our deployment?** No. There is no remote access path. When you want help, you share an export or a screen, under your own process. **How are updates delivered to an air-gapped install?** As a signed bundle with a software bill of materials. Your administrator brings the file in, Foxborne checks the signature against the release key, and nothing installs if it does not match. **What happens if we revoke the encryption key?** Stored originals and exports become unreadable, including to Foxborne. Keys stay in your key management service and rotate on your schedule. **Can we prove a report matches the original logs?** Yes. Every excerpt names its file, its byte offset, row or journal line, and the SHA-256 of the file. Anyone holding the original can check each line. **Does the collector run on our vehicles?** Only if you install it. It is optional: Foxborne works from imported files alone. When installed, it reads, spools and uploads, and it cannot send anything to the flight controller. **Does Foxborne use AI, and does our data go to an AI provider?** It uses AI models in six places, such as proposing a parser for an unfamiliar log and checking a report’s figures against the records. They run on an inference node inside your deployment and ship in the signed release bundle, so nothing goes to a model provider. They never write evidence or a finding, and without the inference node everything else works the same. **Does Foxborne hold FedRAMP, IL5 or CMMC?** No. Foxborne is software you run inside your own environment, within your own authorization boundary. It holds no FedRAMP authorization and no CMMC certification of its own. ## Pilots 1. **Send one incident.** A failure you have already investigated, with the flight log and whatever companion or ground evidence you kept. 2. **We reconstruct it.** Beside your current tools, on your data, with every claim traced to its source. 3. **Compare the answers.** Where the evidence agrees with your conclusion, where it does not, and what it cannot decide. Request a pilot: https://foxborne.com/contact ## Blog - [Boot time, wall time, arrival time: one incident, three clocks](https://foxborne.com/blog/three-clocks) (2026-09-26): A PX4 flight log counts from boot, the companion journal keeps wall time and the ground station logs arrival. Give each an explicit error bound, and you know which events you can put in order and which you cannot. - [Reading an OOM kill in a journalctl export](https://foxborne.com/blog/reading-an-oom-kill) (2026-09-26): The kernel’s out-of-memory report names the thread that asked, the process it killed and every page it counted. systemd then records the unit’s result, and two settings decide what happens next. - [When BindsTo= takes your MAVLink router down with it](https://foxborne.com/blog/bindsto-mavlink-router) (2026-09-26): A dependency in the unit graph can silence telemetry with no radio fault at all. Here is how to spot one in the journal, and how to prove it on the bench before anyone swaps a radio. - [A telemetry gap is not a radio failure](https://foxborne.com/blog/telemetry-gap-not-radio) (2026-09-26): A hole in the ground receive log proves only that nothing arrived. A stopped router, a radio that left the USB bus and a berm all look the same there, and other logs tell them apart. - [CUI on page one: what 32 CFR 2002 and DoDI 5200.48 ask of an engineering report](https://foxborne.com/blog/cui-on-page-one) (2026-09-26): The government-wide rule asks for the same banner on every page and a designation indicator on the first. DoD practice prints plain CUI at top and bottom and moves the codes and a point of contact into a five-line block. - [IRIG 106 for people who live in PX4 logs](https://foxborne.com/blog/irig-106-for-px4-people) (2026-09-26): A range recorder stamps every packet with a 48-bit counter that ticks at 10 MHz, and it records absolute time as a channel of its own. If you can put a ULog on UTC, you already know the method. - [At jet speed, clock error is distance](https://foxborne.com/blog/jet-speed-clock-error) (2026-09-26): At 170 m/s, a ±40 ms clock bound is ±6.8 m of track; on a quad it is ±0.48 m. Draw it along the track, and you can see when a log cannot say which side of a line an event fell on. - [Why PX4’s 10-second link-loss failsafe fires late](https://foxborne.com/blog/px4-link-loss-timer) (2026-09-26): COM_DL_LOSS_T does not start at the last heartbeat. It starts at the last status report that still carried it, which on a real vehicle adds one and a half to almost four seconds. ## Documentation The documentation at https://docs.foxborne.com/ covers deploying Foxborne, collecting data, clocks and evidence levels, working an incident, reports, the rule catalog and the reference. Every page is also Markdown at https://docs.foxborne.com/md/.md, indexed at https://docs.foxborne.com/llms.txt, with all of it in one file at https://docs.foxborne.com/llms-full.txt.