Hardware in the Loop (HIL)

Every pull request that touches code builds the examples and runs them on real silicon before it can merge. This page documents the rigs that do it, in enough detail to reproduce one.

Two rigs run the CI matrix:

Rig	Config	Runner labels
ci	test/hil/tinyusb.json	self-hosted, X64, hathach, hardware-in-the-loop
hfp	test/hil/hfp.json	self-hosted, Linux, X64, hifiphile

ci is hathach’s rig and is what the rest of this page describes. hfp is a similar VM with a uPD720201 card, hosted by hifiphile.

Bill of materials

Part	Used on ci	Notes
Host PC	Ryzen 9 3900X, MSI MAG B550M MORTAR WIFI, 32 GB	Any x86 with a working IOMMU
USB controllers	4 × Renesas uPD720201 (1912:0014 rev 03) on one PCIe card	SSU SU-U3244-12U: four controllers behind an on-board PCIe switch, 12 ports
Leaf hubs	MCS-92M 7-port USB 2.0 hub board	XH2.54 headers instead of Type-A: sturdier under handling and far tidier to route
Cables	XH2.54 → Type-C / micro-B pigtails	Hub-end pin order: +, D−, D+, −
Debug probes	J-Link, ST-Link, RP2040 debug probe (CMSIS-DAP), WCH-Link, TI ICDI, ESP USB-JTAG	One per board — see Attached boards below
USB fixtures	Per host-capable board: one USB-serial adapter and one USB flash drive	Only for boards that run host/dual tests — see below

One card, four uPD720201 controllers behind a PCIe switch.

One leaf hub: power in, upstream to a root port, seven XH2.54 ports out.

Hub-end XH2.54, board-end USB — Type-C shown, micro-B is the same cable.

Proxmox host

1. BIOS

Enable SVM (or VT-x/VT-d), IOMMU, and Above 4G decoding.

2. Kernel command line

In /etc/default/grub, then update-grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt pcie_acs_override=downstream,multifunction"

pcie_acs_override is required because the card’s four controllers sit behind its own PCIe switch, and that switch does not advertise ACS. Without the override all four land in one IOMMU group and none can be passed through individually. It relaxes DMA isolation between them — fine on a dedicated test rig, not on a shared host. Note it is a Proxmox-kernel patch, not mainline: a stock kernel ignores it silently.

3. Bind the controllers to vfio-pci at boot

/etc/modules:

vfio
vfio_iommu_type1
vfio_pci

/etc/modprobe.d/vfio.conf:

options vfio-pci ids=1912:0014
softdep xhci_pci pre: vfio-pci
softdep xhci_pci_renesas pre: vfio-pci

Bind at boot, ahead of the host’s xhci driver — do not rely on Proxmox’s late binding. If the host ever owns these ports, the constant failed enumerations from the boards keep udev busy past 120 s, udevadm settle times out inside ifupdown2-pre, networking.service is cancelled, and the host comes up with no network.

Then update-initramfs -u -k all, reboot, and check:

lspci -nnk -d 1912:0014 | grep -i 'kernel driver'   # vfio-pci

4. Pass the controllers to the VM

One hostpci entry per controller, not per card — take the BDFs from lspci -nn -d 1912:0014:

qm set <vmid> --machine q35 --cpu host \
    --hostpci0 0000:07:00,pcie=1 --hostpci1 0000:08:00,pcie=1 \
    --hostpci2 0000:09:00,pcie=1 --hostpci3 0000:0a:00,pcie=1

qm config <vmid> should then list all four.

Guest

Debian 13, 16 vCPU, 18 GB RAM.

Renesas firmware

The controllers’ ROM firmware is not reliable under HIL churn: Address Device fails with unexpected setup address command completion code 0x11, and the controller eventually dies outright (xHCI host controller not responding, assume dead). Install Renesas firmware 2.0.2.6, which the kernel loads into the controller at boot.

Do this on the kernel that binds the controllers — with passthrough that is the guest, not the Proxmox host.

1. Download 2.0.2.6 from station-drivers. It arrives as k2026fwup1.exe, a Windows self-extracting installer of 1,895,424 bytes. Verify the firmware it contains, not the installer — the md5 in the next step is the one that matters.

2. Unpack it — despite the name, the firmware inside is called UPDATE.mem:

   7z x k2026fwup1.exe -oupd      # or: cabextract -d upd k2026fwup1.exe
   md5sum upd/UPDATE.mem          # 11b49c68a400564b704c6ef17a0e6c0a, 13012 bytes
   

3. Install it under the name the kernel looks for, and rebuild the initramfs (xhci-pci-renesas lives there):

   sudo install -m 644 upd/UPDATE.mem /lib/firmware/renesas_usb_fw.mem
   sudo update-initramfs -u -k all
   sudo reboot
   

4. Confirm the controller is running it. The first check is the one test/hil/usbtest.py gates its own battery on — anything lower and it refuses to run, failing that board’s usbtest cell:

   sudo setpci -s <bdf> 0x6c.l   # whole dword, must be >= 00202609
   dmesg | grep 'hcc params'     # 0x014051cf = firmware loaded, 0x014050cf = ROM fallback
   

The kernel reloads the firmware on every power cycle, so the file must stay installed — that is what the initramfs step is for. The uPD720202 (1912:0015) takes the same firmware and the same check.

A one-off soft lockup warning in renesas_fw_download_image while the firmware is written is expected — it busy-waits over PCI config space for ~30 s.

Software

Purpose	What ci uses
Build	cmake, ninja-build, and a toolchain per family: gcc-arm-none-eabi, a RISC-V GCC, ESP-IDF
Flashing	Five tools, one per Flasher value — see below
Test harness	pip install -r test/hil/requirements.txt — hidapi, pyserial, esptool
Host-side test tools	dfu-util, mtools, libmtp9, libmtp-runtime, alsa-utils (apt) — the DFU, MSC, MTP and audio tests shell out to these
USB inspection and recovery	pciutils (the usbtest firmware gate), uhubctl (apt), tshark for usbmon capture, testusb from the kernel’s tools/usb/testusb.c

The Flasher column in Attached boards names one of five values; only the ones your own boards use have to be installed. The mapping is not always guessable:

Flasher	Binary
jlink	JLinkExe, from the SEGGER J-Link software
stlink	STM32_Programmer_CLI, from STM32CubeProgrammer — not st-flash
openocd	hathach/openocd branch tinyusb — one build merging the Raspberry Pi (RP2350), WCH and Analog Devices (MAX32) forks, none upstream
esptool	esptool (pip)
lm4flash	lm4flash (apt)

Permissions and tools

sudo cp tools/88-tinyusb.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
# the groups 88-tinyusb.rules assigns; skip any the distro does not have
# (`wireshark` only exists once wireshark-common is installed)
for g in adm dialout plugdev users wireshark; do
  getent group "$g" >/dev/null && sudo usermod -aG "$g" "$USER"
done

Add the vendor rules for the probes you use (J-Link, picotool). uhubctl needs one too and no package ships it — without it every port toggle wants root:

# /etc/udev/rules.d/52-uhubctl.rules - root hubs, plus each hub vendor in the rig
SUBSYSTEM=="usb", ATTR{idVendor}=="1d6b", MODE="0664", GROUP="plugdev"
SUBSYSTEM=="usb", ATTR{idVendor}=="1a40", MODE="0664", GROUP="plugdev"
SUBSYSTEM=="usb", ATTR{idVendor}=="045b", MODE="0664", GROUP="plugdev"

Flasher CLIs and toolchains must be reachable from non-interactive shells — neither the Actions runner nor hil_ci.sh sources a login profile. Keep them in ~/.local/bin and ~/bin (symlinks are fine) and add both to the runner’s .path.

pciutils and passwordless sudo are hard requirements, not conveniences: test/hil/usbtest.py shells out as sudo -n for setpci, modprobe, dmesg and testusb, and exits outright if it cannot read the host controller’s firmware version. helper/hil_pool_check.py gates recovery on the same sudo -n plus .claude/skills/usb-kernel-recover/scripts/usb_recover.sh being present; without both it cannot re-authorize a wedged probe’s port and files the board flash-failed instead.

The usbtest battery additionally needs testusb built from the kernel tools and CONFIG_USB_TEST=m available.

USB topology

One 7-port hub per uPD720201 root port. Never chain hubs.

Each controller presents four root ports (on both its USB 2 and USB 3 root hubs); the card brings 12 of those 16 out to connectors. Hang exactly one leaf hub on a root port.

Boards are grouped into storage boxes, each holding two leaf hubs: one carries only debug probes, the other only the boards under test. Keeping them apart is what makes recovery tractable — a DUT re-enumerates constantly and can wedge its hub, while the probes stay on a bus that never moves, so the probe you need to reset a hung board is still there when you reach for it.

One box: boards, their probes, and the two leaf hubs serving them.

Boards that run host or dual tests additionally need a USB peripheral plugged into the board’s own USB port — a USB-serial adapter and/or a flash drive for the host stack to enumerate. Ten ci boards have these, recorded as dev_attached in the rig config and matched by exact VID:PID and serial, so a substitute part means updating the config. The two Espressif boards also use a TS3USB30 mux to drive device and host tests through one connector.

Why the rule matters:

• Bandwidth. Every leaf hub gets its own 480 Mbit uplink to the controller. Chaining puts a second hub’s whole subtree behind one of those uplinks, and the usbtest battery saturates whatever it is given.

• Blast radius. A board that wedges its hub costs seven ports, not the rig.

• Scheduling. hil_test.py budgets flashing and usbtest concurrency per host controller (test/hil/helper/hil_lock.py: FLASH_PARALLEL, USBTEST_PARALLEL), which only means anything when a controller’s set of devices is fixed.

Bus numbers are not stable across reboots or recabling, so nothing in the harness addresses a board by bus path. Boards are identified by the MCU’s unique ID and probes by their serial, both recorded in the rig config — which is why every HIL board must implement board_get_unique_id().

Attached boards

Roles come from each board’s tests entry; Flasher is the tool that programs it. Both files are the source of truth — this table is generated from them.

ci rig

27 boards, from test/hil/tinyusb.json.

Board	Roles	Flasher	Variants	Note
frdm_k64f	host	jlink
ek_tm4c123gxl	device	lm4flash
espressif_p4_function_ev	device, host	esptool	espressif_p4_function_ev, espressif_p4_function_ev-DMA	Use TS3USB30 mux to test both device and host
espressif_s3_devkitm	device, host	esptool	espressif_s3_devkitm, espressif_s3_devkitm-DMA	Use TS3USB30 mux to test both device and host
feather_nrf52840_express	device	jlink
max32666fthr	device	openocd
metro_m4_express	device, dual	jlink	metro_m4_express	pl23x; audio_test_freertos skipped: samd51 iso-IN capture fails (arecord EIO)
lpcxpresso11u37	device	jlink
lpcxpresso55s28	device	jlink
ra4m1_ek	device	jlink
raspberry_pi_pico	device, host, dual	openocd	raspberry_pi_pico
raspberry_pi_pico_w	host	openocd		Test native host
raspberry_pi_pico2	host	openocd
adafruit_fruit_jam	device, host, dual	openocd
stm32f072disco	device	jlink		2x16 access scheme with 1KB USB SRAM
stm32f407disco	device	jlink
stm32f723disco	device, host	jlink	stm32f723disco, stm32f723disco-DMA	Device port0 FS (slave only), Host port1 HS with DMA
stm32h743nucleo	device	stlink	stm32h743nucleo, stm32h743nucleo-DMA
stm32g0b1nucleo	device	stlink		32-bit scheme, 2KB USB SRAM
stm32l476disco	device	jlink
stm32u083nucleo	device	stlink
nanoch32v203	device	openocd	nanoch32v203-fsdev, nanoch32v203-usbfs
ch32v103r_r1_1v0	device	openocd
ch32v307v_r1_1v0	device	openocd	ch32v307v_r1_1v0-usbhs, ch32v307v_r1_1v0-usbfs
ch582m_evt	device	openocd
mimxrt1064_evk	device, host, dual	jlink
nrf54lm20dk	device	jlink		board new to HIL: audio_test_freertos never reaches dcd_init (FreeRTOS itself runs; cdc_msc_freertos and usbtest pass) - example-level issue on nRF54L, fix separately

hfp rig

3 boards, from test/hil/hfp.json.

Board	Roles	Flasher	Variants	Note
stm32l412nucleo	device	stlink
stm32f746disco	device	stlink	stm32f746disco, stm32f746disco-DMA
lpcxpresso43s67	device	jlink

How CI runs the tests

1. hil-build and hil-build-esp build the examples on GitHub-hosted runners and upload the binaries as artifacts.

2. hil-tinyusb runs on the self-hosted rigs, downloads those artifacts and calls test/hil/hil_test.py, which flashes each board and runs its tests. Espressif boards run in hil-tinyusb-esp, gated on the slower ESP-IDF build, and hil-hfp-iar builds with IAR inside the job.

3. On pull requests, tools/ci_select.py narrows the run to the boards a diff can affect — and each board’s build to the examples its tests need — falling open to the full matrix when it cannot tell. The same pass scopes the build matrix.

4. Each board is arbitrated by a kernel flock in /tmp/tinyusb-hil-locks/, so interactive work and CI can share the rig without colliding.

5. Each rig job uploads its report as an artifact; pr_comment.yml downloads them and posts the combined tables onto the pull request.

From a development PC, the same run can be driven remotely. REMOTE and CONFIG default to ci, so point them at your own:

REMOTE=myrig.lan CONFIG=$PWD/test/hil/local.json bash test/hil/hil_ci.sh -b <board>

Gotchas

• The Renesas firmware is not optional. On ROM firmware these controllers fail Address Device and eventually die under test churn.

• Port power is logical only. uhubctl “off” on these controllers drops D+/D− but leaves VBUS hot — boards stay powered and running. Real per-port power switching needs the controller’s PPON pins wired to load switches, which the card omits.

• Use uhubctl -S on root ports. Without it, uhubctl writes sysfs disable, which takes the root hub’s lock — and if anything in that subtree is in D state it blocks there, leaving the whole bus untouchable. -S forces the libusb path instead, which is why usb_recover.sh root-cycle uses it. Resetting the board through its debug probe is the surer cure, but a wedged probe has none, so the port-side drop is the only lever left there.

• Park firmware must busy-spin, never wfe/wfi. A parked core in a low-power state can make SWD unreachable and leave the board needing recovery.

• Most “7-port” hubs are two 4-port hubs in series. Commodity 7-port hubs commonly cascade two controllers internally — three ports on the first, four behind a second. lsusb -t tells you which you bought: a single-tier hub appears as one device with seven ports, a cascaded one shows a hub inside a hub. Every hub on ci sits directly under a root port and reports maxchild=7.

• Size the hub supplies. Boards take VBUS from the leaf hub, so a seven-board hub on an undersized supply browns out under load.

A minimal rig

None of the above is a prerequisite. The VM, the uPD720201 cards and the leaf hubs are what let one machine hold 27 boards and recover them unattended — the harness itself runs fine against boards plugged straight into a development PC’s own USB ports, on whatever xHCI that PC already has. All it takes is the boards, their debug probes, and a test/hil/local.json describing them in the same shape as tinyusb.json.

Host-side prerequisites, beyond a cross toolchain:

python3 tools/get_deps.py <family>            # MCU SDKs for your boards
pip install -r test/hil/requirements.txt      # hidapi, pyserial, esptool
sudo apt install cmake ninja-build uhubctl \
                 dfu-util mtools libmtp9 libmtp-runtime alsa-utils

cmake and ninja-build are needed by any run and uhubctl by recovery; the rest only by the tests that shell out to them, so dropping one just fails the DFU, MSC, MTP or audio cells on an otherwise healthy rig. test/hil/requirements.txt names those at the top, along with iperf for the device/net_lwip_* tests, which are off in the default matrix.

Only two of this page’s host-controller concerns carry over. test/hil/usbtest.py refuses a DUT behind a MosChip MCS9990 (9710:9990) outright, and it applies the Renesas firmware check only when the DUT really is behind a uPD720201/02 — on a stock Intel or AMD xHCI there is nothing to install, and pciutils is only needed for that check.

cd examples && cmake --preset <board> && cmake --build --preset <board>
cd .. && python3 test/hil/hil_test.py -B examples test/hil/local.json