I’m Touched: Screen Firmware

After I wired up frame sync, the tearing came back, but only on the bottom half of the screen, and only when the content moved. A static JPEG sent on repeat was clean. The live-rendered torus tore, every few frames, always in the lower rows.

The obvious suspect was a race: maybe duplicate sync packets were making the receive path re-fill a buffer mid-decode. Logging killed that theory (one “new frame” per frame id, no duplicates). The real cause was arithmetic, not concurrency. TouchDesigner sends at 20 fps; each device decodes and displays at about 16. The sync logic waited for a sync packet matching exactly the frame it had just decoded. But by the time frame N finished decoding, the syncs for N+1 through N+3 had already arrived and overwritten the “latest sync” register, so the wait for “exactly N” could never succeed, and every single frame burned its full 50 ms timeout. That pushed decode so far behind the incoming stream that the reassembly ring wrapped and the WiFi task started overwriting a JPEG the decoder was still reading. Hence the tearing, in whichever rows decode hadn’t reached yet.

The fix is a signed 16-bit subtraction:

// Signed 16-bit delta handles frame_id wraparound.  If sync is already at
// or past us, present immediately — waiting for a sync that already
// happened would just add latency and push decode into buffer-overrun.
const int16_t diff = (int16_t)(s_last_frame_id - frame_id);
if (diff >= 0) {
    display_driver::present();
    return;
}

“Sync at-or-past us means go” turns the slow consumer’s lateness into zero extra cost, and the signed cast makes the comparison survive frame-id wraparound (at 20 fps, every ~55 minutes). This made the difference between “works” and “tears constantly,” and the same hazard applies across the firmware: any buffer a fast producer shares with a slow consumer can overrun the same way.

The shape of the firmware

Each screen module is an ESP32-S3 (dual-core, 240 MHz, 16 MB flash, 8 MB octal PSRAM) driving a 240×320 ILI9341 panel over 60 MHz SPI. The firmware is pure ESP-IDF 5.4 with no Arduino anywhere, organized as five tasks with a strict core split:

Core 0 is everything network, sharing the core with the WiFi driver. Core 1 does nothing but decode and push pixels, so the hot path never competes with the radio.

Memory placement

The S3 has ~330 KB of usable internal SRAM and 8 MB of PSRAM, and where each buffer lives is most of the design:

• The RGB565 back buffer (150 KB) is internal SRAM, allocated MALLOC_CAP_DMA | MALLOC_CAP_INTERNAL. It’s written by the decoder and read by the SPI DMA engine; putting it in PSRAM costs 30-40% of DMA throughput when the decoder is also touching PSRAM.
• The JPEG reassembly ring (4 × 32 KB) is PSRAM. Fragments are written sequentially and read sequentially by the decoder, an access pattern PSRAM handles fine, and 128 KB of internal SRAM was never on the table.

The slot count comes from what can be in flight at once: one buffer being filled by udp_rx, up to two sitting in the decode queue (its depth is 2), and one being decoded. 1 + 2 + 1 = 4; with exactly four slots the fill pointer can’t lap anything in use. Raise the queue depth and the slot count must follow; the invariant is written down next to both numbers.

The receive path itself is defensive in small, load-bearing ways: a uint32_t bitmask tracks received fragments (which is where the protocol’s 32-fragment / 32 KB frame ceiling comes from), duplicate fragments are dropped before they touch memory, a frame whose first fragment is older than 100 ms is abandoned so a half-arrived frame can’t wedge the pipeline, and a full decode queue drops the frame rather than blocking the receive loop, with each drop counted in the heartbeat’s dropped-frames field instead of vanishing silently. (That counter originally only tracked decode errors; wiring queue-full drops into it is what made the telemetry actually useful for tuning.)

Sixty milliseconds

Sustained throughput is ~16 fps per device.

The two stages serialize because there’s a single back buffer: DMA reads the same memory the next decode wants to write, so present() ends with waitDMA(). A second 150 KB back buffer would let them overlap (max(30, 30) instead of 30 + 30, a ~33 fps ceiling), and at 300 KB of 330 KB internal SRAM it would just barely fit. It’s deliberately not done. 16 fps is plenty for a slowly rotating object, and the mechanical sphere needed the hours more.

Oof the 4:2:0 woes

Early frames decoded with strange, content-dependent artifacts: colored horizontal stripes in flat gray regions, ghost rectangles to the right of sharp text. Solid-color JPEGs were perfect. The diagnostic ladder, in order:

1. Fill the back buffer directly with color bands, skip the decoder entirely, DMA it out → clean. The display path is innocent.
2. Log every decode callback’s coordinates → 45 callbacks for a 320×240 image, all sane. Placement is correct.
3. Replace the callback’s memcpy with a memset to a position-coded color → three clean vertical columns. The copy is clean, too.
4. So the decoded pixels themselves are wrong. Same image at quality 95 (which encoders emit as 4:4:4) → clean. Same image with 4:2:0 subsampling forced → glitches.

It turns out JPEGDEC v1.8.4’s 4:2:0 chroma path mis-decodes on this target. The fix lives on the TouchDesigner side, forcing 4:4:4 at encode time, and costs ~30% larger files, well inside the 32 KB frame ceiling.

Twelve devices, one binary

Every board runs the identical binary; what differs is two bytes in NVS (device_index and screen_rotation), written once over the same USB-C cable that powers the board (the S3’s console runs on native USB-Serial-JTAG). A tiny stdin task speaks two commands, prov <idx> <rot> and get, and a host-side script drives it:

$ ./tools/provision_device.py --port /dev/cu.usbmodem21101 --device-index 7 --rotation 1
OK: stored idx=7 rot=1 mac=44:1B:F6:CE:5A:88; rebooting

The MAC in the reply goes straight into the router’s DHCP reservation table, and the heartbeat listener discovers the resulting IP automatically. One ordering lesson embedded in app_main: the provisioning console starts before WiFi, because the WiFi bring-up can block for its full 15-second association window, and a board that can’t join the network is exactly the board you most need a working console on.

Power, and the battery badge

Each module runs on a Li-Po behind a panel toggle, so the firmware grew power features late. Hold BOOT for two seconds and the device deep-sleeps (backlight off first, since it’s the dominant ~79 mA of the load) so USB-C can charge the battery without the device drawing 150+ mA. BOOT wakes it again.

The battery badge is my favorite small feature: a battery indicator drawn on-screen only while on external power, so a charging module reports its state and an installed one keeps a clean image. Detecting “on external power” without a dedicated sense pin took two mechanisms. If a USB host is enumerated, usb_serial_jtag_is_connected() gives a definitive answer immediately. On a dumb wall brick there’s no enumeration, so the firmware watches the battery voltage’s 60-second trend and calls a sustained rise “charging.” A plain voltage threshold wouldn’t work, because a full Li-Po rests above 4.1 V for hours after unplugging and would pin the badge on.