The Camera Rig

There are twelve cameras in the TouchDesigner project, and every one of them is standing where a screen is: physical screen i sits at position P_i on the physical video sphere facing the center, virtual camera i sits at P_i on a virtual sphere aimed at the origin, and if those two facts stay true, the object at the origin appears to float inside the sculpture. The sender, the calibration mode, and the heartbeat listener all exist to keep that correspondence true.

The rig

The cameras are placed by script, not by hand. Twelve cameraCOMPs sit at icosahedron vertices, scaled onto a sphere of radius 2.5, each with a matching 320×240 renderTOP:

SPHERE_R = 2.5
PHI   = (1.0 + math.sqrt(5.0)) / 2.0
SCALE = SPHERE_R / math.sqrt(1.0 + PHI * PHI)

for i, (rx, ry, rz) in enumerate(raw):   # the 12 (0, ±1, ±φ) permutations
    cam = p.op(f'cam_{i}') or p.create(cameraCOMP, f'cam_{i}')
    cam.par.tx, cam.par.ty, cam.par.tz = rx * SCALE, ry * SCALE, rz * SCALE
    cam.par.lookat = '/project1/target'
    cam.par.fov    = 45
    cam.par.upx, cam.par.upy, cam.par.upz = 0, 1, 0

There’s no pole handling. The spec has a whole degenerate-case branch for cameras near the ±Y axis, where a world-Y up vector becomes parallel to the view direction and the math falls apart. It never runs, because the icosahedron has no axial poles; the largest normalized |y| among its vertices is about 0.851. Every camera gets up = (0, 1, 0). That simplification was bought back in the hardware post, with the choice of solid.

The atlas that wasn’t

The spec’s plan was tidy: composite all twelve renders into one atlas TOP, read it with a single numpyArray() call per frame, slice tiles out in NumPy. I tried. TouchDesigner’s Layout TOP, configured with outputresolution = custom 1600×480, fit = nativeres, align = gridrows, and maxcols = 5, produced six inputs scattered in a 2×3 arrangement of centered tiles instead of the 5-wide grid those parameters describe. outputresolution = useinput collapsed everything to one overlapping 320×240; fit = fill squashed it. After two diagnostic passes and about thirty minutes, I stopped debugging the node and changed the sender to read each render_N TOP directly.

Reading them separately costs nothing: the same 921,600 pixels cross the GPU→CPU boundary each frame whether they arrive as one atlas or as twelve tiles, and the read time is dominated by that transfer, not by per-call overhead. If someone knows the Layout TOP incantation I missed, I’d genuinely like to hear it, though I’d still ship the per-TOP version because it’s less code.

The sender

The sender runs on every frame-start, rate-limited to 20 fps. Per device: read the render, flip it, quantize, JPEG-encode, fragment, send. Then one sync broadcast for the whole fleet:

for device_id, top_name in enumerate(DEVICE_TOPS):
    ip = hb.get_ip(device_id)
    if ip is None:
        continue                      # no heartbeat from this device yet
    arr = op(top_name).numpyArray(delayed=True)
    if arr is None:
        continue
    arr  = np.flipud(arr)             # OpenGL bottom-left → top-left
    tile = (arr[:, :, :3] * 255).astype(np.uint8)
    jpeg = _encode_jpeg_444(tile)
    if jpeg is not None:
        _send_fragmented(jpeg, fid, device_id, ip)
_send_sync(fid)

Three lines carry most of the weight:

• numpyArray(delayed=True) returns the previous cook’s framebuffer instead of forcing a synchronous re-render. The sender never stalls TD’s render loop waiting for pixels; it’s always exactly one frame behind, which nothing downstream can perceive.
• np.flipud(arr) flips the frame, because TOP framebuffers are OpenGL with a bottom-left origin and the panels want top-left. One central flip on the host beats twelve per-device firmware opinions about which way is up.
• _encode_jpeg_444 forces 4:4:4 chroma subsampling (cv2.IMWRITE_JPEG_SAMPLING_FACTOR = 0x111111). Encoders default to 4:2:0 for good bandwidth reasons, but the decoder on the ESP32 mis-renders 4:2:0; the firmware post walks the diagnostic ladder that pinned it down. The cost is ~30% larger files. At quality 55 a typical tile runs 4-10 KB against a 32 KB protocol ceiling, roughly 3× headroom.

Fragmentation is a plain 8-byte header, struct.pack('<HBBHBB', frame_id, i, frag_count - 1, total, flags, device_id), matching the firmware’s packed C struct byte for byte, with at most 32 fragments of 1024 bytes each.

Everything runs inside the frame

The rule that shaped all of this code: TouchDesigner Python is game-engine code. You’re always inside the frame, on the main thread, and everything you do delays the next cook. Three places that rule bit:

1. No threads. All op() access is main-thread-only, so the tempting “spawn a listener thread” pattern won’t work. The heartbeat listener instead drains a non-blocking socket inline every frame, bounded to at most 64 packets per drain so a burst can’t blow the frame budget.
2. No sleeps. The spec calls for sending each sync packet 3× with 2 ms spacing for redundancy. time.sleep(0.002) on TD’s main thread freezes the UI, and at 20 fps those sleeps would eat 4 ms (8% of a 50 ms frame). The sender broadcasts sync once; the firmware’s sync handling is permissive enough that this works. (If multi-device desync ever appears, the fix is run(..., delayFrames=1): scheduled follow-ups instead of sleeps.)
3. No blocking reads. That’s delayed=True again.

The heartbeat is the router

Each device broadcasts a 1 Hz heartbeat (device id, last frame seen, frames completed and dropped in the last second). The listener does two things with it. It populates a status table, flagging any device silent for 3 seconds as offline. More importantly, it captures each packet’s source IP into a device_id → IP map that the sender uses for routing.

That second use replaced the spec’s static-IP plan. Devices come up on DHCP, announce themselves, and the sender starts targeting whatever address they actually have, within a second of boot. There’s no per-device network config to maintain, and a re-plugged board keeps working even if its address changed. The spec’s static addressing still exists behind a firmware build flag for the eventual dedicated-AP installation, but the heartbeat map is what bring-up actually ran on.

Calibration mode

On a flat video wall, device 3 in the wrong slot is obvious. On a sphere there’s no canonical top-left (the spec is blunt that confusing device IDs during assembly is extremely easy), so the sender has a calibration mode, toggled by a custom parameter on a controls COMP. Instead of the live render, each device gets a generated tile: its ID as a big white-on-black digit, a background hue unique to that device (hue = id/N × 360; mind that OpenCV’s hue axis is 0-179 half-degrees), and an up-arrow to verify the physical mounting rotation matches the virtual camera’s up vector. Tiles are cached after first build, so calibration costs the same ~6 ms per frame as live rendering.

If the walk-around turns up a mismatch, there are three fixes, in increasing order of commitment: swap the physical module, re-provision its NVS device_index, or set its screen_rotation. Non-90° rolls can only be fixed here, on the camera’s up vector; the panel can’t rotate pixels by arbitrary angles.

What’s still open

The calibration session on the fully assembled sphere hasn’t happened yet; the mode is built and waiting for the walk-around. Until then the proof is the torus on twelve desks: one send_frame() loop showing the same object from twelve angles.

Part 5 of the video-sphere series. Previous: The Hardware Revisions · Next: The Screen Firmware.