The Screen Firmware

After I wired up frame sync, the tearing came back, but only on the bottom half of the screen, and only when the content moved. A static JPEG sent on repeat was clean. The live-rendered torus tore, every few frames, always in the lower rows.

The obvious suspect was a race: maybe duplicate sync packets were making the receive path re-fill a buffer mid-decode. Logging killed that theory (one “new frame” per frame id, no duplicates). The real cause was arithmetic, not concurrency. TouchDesigner sends at 20 fps; each device decodes and displays at about 16. The sync logic waited for a sync packet matching exactly the frame it had just decoded. But by the time frame N finished decoding, the syncs for N+1 through N+3 had already arrived and overwritten the “latest sync” register, so the wait for “exactly N” could never succeed, and every single frame burned its full 50 ms timeout. That pushed decode so far behind the incoming stream that the reassembly ring wrapped and the WiFi task started overwriting a JPEG the decoder was still reading. Hence the tearing, in whichever rows decode hadn’t reached yet.

The fix is a signed 16-bit subtraction:

// Signed 16-bit delta handles frame_id wraparound.  If sync is already at
// or past us, present immediately — waiting for a sync that already
// happened would just add latency and push decode into buffer-overrun.
const int16_t diff = (int16_t)(s_last_frame_id - frame_id);
if (diff >= 0) {
    display_driver::present();
    return;
}

“Sync at-or-past us means go” turns the slow consumer’s lateness into zero extra cost, and the signed cast makes the comparison survive frame-id wraparound (at 20 fps, every ~55 minutes). This was the difference between “works” and “tears constantly,” and the same hazard applies across the firmware: any buffer a fast producer shares with a slow consumer can overrun the same way.

The shape of the firmware

Each screen module is an ESP32-S3 (dual-core, 240 MHz, 16 MB flash, 8 MB octal PSRAM) driving a 240×320 ILI9341 panel over 60 MHz SPI. The firmware is pure ESP-IDF 5.4 with no Arduino anywhere (a decision explained below), organized as five tasks with a strict core split:

Core 0 is everything network, sharing the core with the WiFi driver. Core 1 does nothing but decode and push pixels, so the hot path never competes with the radio.

Memory placement

The S3 has ~330 KB of usable internal SRAM and 8 MB of PSRAM, and where each buffer lives is most of the design:

• The RGB565 back buffer (150 KB) is internal SRAM, allocated MALLOC_CAP_DMA | MALLOC_CAP_INTERNAL. It’s written by the decoder and read by the SPI DMA engine; putting it in PSRAM costs 30-40% of DMA throughput when the decoder is also touching PSRAM, per the spec’s measurements.
• The JPEG reassembly ring (4 × 32 KB) is PSRAM. Fragments are written sequentially and read sequentially by the decoder, an access pattern PSRAM handles fine, and 128 KB of internal SRAM was never on the table.

The slot count comes from what can be in flight at once: one buffer being filled by udp_rx, up to two sitting in the decode queue (its depth is 2), and one being decoded. 1 + 2 + 1 = 4; with exactly four slots the fill pointer can’t lap anything in use. Raise the queue depth and the slot count must follow; the invariant is written down next to both numbers.

The receive path itself is defensive in small, load-bearing ways: a uint32_t bitmask tracks received fragments (which is where the protocol’s 32-fragment / 32 KB frame ceiling comes from), duplicate fragments are dropped before they touch memory, a frame whose first fragment is older than 100 ms is abandoned so a half-arrived frame can’t wedge the pipeline, and a full decode queue drops the frame rather than blocking the receive loop, with each drop counted in the heartbeat’s dropped-frames field instead of vanishing silently. (That counter originally only tracked decode errors; wiring queue-full drops into it is what made the telemetry actually useful for tuning.)

Sixty milliseconds, itemized

Sustained throughput is ~16 fps per device, and the budget is embarrassingly legible:

The two stages serialize because there’s a single back buffer: DMA reads the same memory the next decode wants to write, so present() ends with waitDMA(). A second 150 KB back buffer would let them overlap (max(30, 30) instead of 30 + 30, a ~33 fps ceiling), and at 300 KB of 330 KB internal SRAM it would just barely fit. It’s deliberately not done. 16 fps is plenty for a slowly rotating object, and the mechanical sphere needed the hours more.

Two benched details hiding in that table. First, 60 MHz SPI, not the 80 the IOMUX pins support: at 80 MHz the panel tore intermittently, once every 3-4 seconds, which is classic edge-of-signal-integrity behavior on a module whose traces aren’t impedance-controlled. The ~5 ms the higher clock would save isn’t worth a sculpture that glitches on camera. Second, the raw 60 MHz transfer math says 20.5 ms; the measured 30 includes command overhead, DMA setup, and LovyanGFX bookkeeping.

The 4:2:0 hunt

Early frames decoded with strange, content-dependent artifacts: colored horizontal stripes in flat gray regions, ghost rectangles to the right of sharp text. Solid-color JPEGs were perfect. The diagnostic ladder, in order:

1. Fill the back buffer directly with color bands, skip the decoder entirely, DMA it out → clean. The display path is innocent.
2. Log every decode callback’s coordinates → 45 callbacks for a 320×240 image, all sane. Placement is correct.
3. Replace the callback’s memcpy with a memset to a position-coded color → three clean vertical columns. The copy is clean, too.
4. So the decoded pixels themselves are wrong. Same image at quality 95 (which encoders emit as 4:4:4) → clean. Same image with 4:2:0 subsampling forced → glitches.

Verdict: JPEGDEC v1.8.4’s 4:2:0 chroma path mis-decodes on this target. The fix lives on the TouchDesigner side, forcing 4:4:4 at encode time, and costs ~30% larger files, well inside the 32 KB frame ceiling.

A related half-hour: the first decoded frames had colors that were “almost right but rotated.” Red came out dark blue-teal. That’s RGB565 byte order. The host CPU is little-endian, so red 0xF800 sits in memory as {0x00, 0xF8}; the ILI9341 reads the high byte first and sees 0x00F8, which is a dim blue-green. JPEGDEC’s setPixelType(RGB565_BIG_ENDIAN) makes it write panel order directly, and the DMA pushes bytes as-is.

Escaping Arduino

The spec suggested framework = arduino, espidf (Arduino as an ESP-IDF component). That path collapsed under its own dependencies: arduino-esp32’s manifest transitively pulls ~26 managed components (esp_insights, esp_rainmaker, esp-zigbee-lib, an MP3 decoder…), and esp_insights tripped a CMake path-construction bug in the build glue:

*** [.pio/build/device_runtime/.pio/build/device_runtime/https_server.crt.S.o]
    Source `.pio/build/device_runtime/https_server.crt.S' not found

Note the doubled path. After failing to cleanly exclude the component two different ways, I stopped fighting and went pure ESP-IDF: setup()/loop() became app_main(), WiFi.h became esp_wifi_*, and the binary shrank from ~780 KB to ~280 KB. The one casualty was JPEGDEC’s packaging: its metadata declares an Arduino-only dependency it doesn’t actually need, which makes PlatformIO’s dependency finder skip it on pure-IDF builds. Vendoring the six source files into lib/ solved it. The rule of thumb I keep now: default to pure IDF, and vendor any library with eccentric metadata rather than fighting the resolver.

Twelve devices, one binary

Every board runs the identical binary; what differs is two bytes in NVS (device_index and screen_rotation), written once over the same USB-C cable that powers the board (the S3’s console runs on native USB-Serial-JTAG). A tiny stdin task speaks two commands, prov <idx> <rot> and get, and a host-side script drives it:

$ ./tools/provision_device.py --port /dev/cu.usbmodem21101 --device-index 7 --rotation 1
OK: stored idx=7 rot=1 mac=44:1B:F6:CE:5A:88; rebooting

The MAC in the reply goes straight into the router’s DHCP reservation table, and the heartbeat listener discovers the resulting IP automatically. One ordering lesson embedded in app_main: the provisioning console starts before WiFi, because the WiFi bring-up can block for its full 15-second association window, and a board that can’t join the network is exactly the board you most need a working console on.

WiFi itself had two problems worth naming. There were two access points in range broadcasting the same SSID with different subnets, so the firmware scans, logs every matching BSSID, and hard-locks to the one that owns its network: association by MAC address, not by name. And the retry policy is to never give up. The comment in the handler says why better than I can summarize it: “The AP may come back, the user may walk closer, or a different ‘touch-me’ may temporarily out-shout ours.”

Power, and the battery badge

Each module runs on a Li-Po behind a panel toggle, so the firmware grew power features late. Hold BOOT for two seconds and the device deep-sleeps (backlight off first, since it’s the dominant ~79 mA of the load) so USB-C can charge the battery without the device drawing 150+ mA. BOOT wakes it again.

The battery badge is my favorite small feature: a battery indicator drawn on-screen only while on external power, so a charging module reports its state and an installed one keeps a clean image. Detecting “on external power” without a dedicated sense pin took two mechanisms. If a USB host is enumerated, usb_serial_jtag_is_connected() gives a definitive answer immediately. On a dumb wall brick there’s no enumeration, so the firmware watches the battery voltage’s 60-second trend and calls a sustained rise “charging.” A plain voltage threshold wouldn’t work, because a full Li-Po rests above 4.1 V for hours after unplugging and would pin the badge on.

Aggregate check

Twelve devices × 20 fps × ~8 KB per frame ≈ 1.9 MB/s ≈ 15 Mbps, comfortable inside a 2.4 GHz 802.11n HT20 channel’s practical ~25 Mbps. The spec’s dual-AP contingency starts at 16 screens; at twelve, one AP carries it.

Still open: the second back buffer (33 fps is sitting right there), RSSI in the heartbeat so the assembled sphere can be surveyed for weak spots, and the move to a dedicated, isolated AP with the static-IP build flag flipped. Nothing on that list blocks the sculpture, and everything on it is written down in the status doc.

Part 6 of the video-sphere series. Previous: The Camera Rig · Next: The Whole System.