I’m Touched: The TouchDesigner Camera Rig

There are twelve cameras in the TouchDesigner project, and every one of them is standing where a screen is: physical screen i sits at position P_i on the physical video sphere facing the center, virtual camera i sits at P_i on a virtual sphere aimed at the origin, and if those two facts stay true, the object at the origin appears to float inside the sculpture. The sender, the calibration mode, and the heartbeat listener all exist to keep that correspondence true.

The rig

The cameras are placed by script, not by hand. Twelve cameraCOMPs sit at icosahedron vertices, scaled onto a sphere of radius 2.5, each with a matching 320×240 renderTOP:

SPHERE_R = 2.5
PHI   = (1.0 + math.sqrt(5.0)) / 2.0
SCALE = SPHERE_R / math.sqrt(1.0 + PHI * PHI)

for i, (rx, ry, rz) in enumerate(raw):   # the 12 (0, ±1, ±φ) permutations
    cam = p.op(f'cam_{i}') or p.create(cameraCOMP, f'cam_{i}')
    cam.par.tx, cam.par.ty, cam.par.tz = rx * SCALE, ry * SCALE, rz * SCALE
    cam.par.lookat = '/project1/target'
    cam.par.fov    = 45
    cam.par.upx, cam.par.upy, cam.par.upz = 0, 1, 0

There’s no pole handling. A camera near the ±Y axis would need it — there, a world-Y up vector becomes parallel to the view direction and the math falls apart — but no camera ever sits there, because the icosahedron has no axial poles; the largest normalized |y| among its vertices is about 0.851. Every camera gets up = (0, 1, 0). The choice of solid paid for that simplification.

The sender

The sender runs on every frame-start, rate-limited to 20 fps. Per device: read the render, flip it, quantize, JPEG-encode, fragment, send. Then one sync broadcast for the whole fleet:

for device_id, top_name in enumerate(DEVICE_TOPS):
    ip = hb.get_ip(device_id)
    if ip is None:
        continue                      # no heartbeat from this device yet
    arr = op(top_name).numpyArray(delayed=True)
    if arr is None:
        continue
    arr  = np.flipud(arr)             # OpenGL bottom-left → top-left
    tile = (arr[:, :, :3] * 255).astype(np.uint8)
    jpeg = _encode_jpeg_444(tile)
    if jpeg is not None:
        _send_fragmented(jpeg, fid, device_id, ip)
_send_sync(fid)

Everything runs inside the frame

The rule that shaped all of this code: TouchDesigner Python is game-engine code. You’re always inside the frame, on the main thread, and everything you do delays the next cook. Three places that rule bit:

1. No threads. All op() access is main-thread-only, so the tempting “spawn a listener thread” pattern won’t work. The heartbeat listener instead drains a non-blocking socket inline every frame, bounded to at most 64 packets per drain so a burst can’t blow the frame budget.
2. No sleeps. The original plan was to send each sync packet 3× with 2 ms spacing for redundancy. time.sleep(0.002) on TD’s main thread freezes the UI, and at 20 fps those sleeps would eat 4 ms (8% of a 50 ms frame). The sender broadcasts sync once; the firmware’s sync handling is permissive enough that this works. (If multi-device desync ever appears, the fix is run(..., delayFrames=1): scheduled follow-ups instead of sleeps.)
3. No blocking reads. That’s delayed=True again.

The heartbeat as a router

Each device broadcasts a 1 Hz heartbeat (device id, last frame seen, frames completed and dropped in the last second). The listener does two things with it. It populates a status table, flagging any device silent for 3 seconds as offline. More importantly, it captures each packet’s source IP into a device_id → IP map that the sender uses for routing.

That second use replaced the original static-IP plan. Devices come up on DHCP, announce themselves, and the sender starts targeting whatever address they actually have, within a second of boot. There’s no per-device network config to maintain, and a re-plugged board keeps working even if its address changed. Static addressing still exists behind a firmware build flag for the eventual dedicated-AP installation, but the heartbeat map is what bring-up actually ran on.

Calibration mode

On a flat video wall, device 3 in the wrong slot is obvious. On a sphere there’s no canonical top-left, and confusing device IDs during assembly is extremely easy, so the sender has a calibration mode, toggled by a custom parameter on a controls COMP. Instead of the live render, each device gets a generated tile: its ID as a big white-on-black digit, a background hue unique to that device (hue = id/N × 360; mind that OpenCV’s hue axis is 0-179 half-degrees), and an up-arrow to verify the physical mounting rotation matches the virtual camera’s up vector. Tiles are cached after first build, so calibration costs the same ~6 ms per frame as live rendering.