| Creative Coding

Three.js – Week Two

Cover Image for Three.js – Week Two

Overview

This week I wanted to learn more about camera movement and effects in three.js. I also wanted to experiment with bringing in outside data into the animation. M and I are working on a project to visualize group dynamics in a small room, so I started with laying data on top of a video feed using regular canvas elements. The ”levers” we were looking for in order to control the visualization were person count, average group X and Y coordinates, and a proximity or unity factor describing how close the audience was in relation to one another.

After I was able to successfully get these paramaters from a video feed, I started experimenting what an output would look like using three.js.

Input

Sourcing the data from a video feed occurred in a series of measurements that build off of one another. Throughout the process, I drew directly onto a canvas with regular canvas drawing tools to ensure I was describing the data correctly.

Posenet

The first representation of the group comes the Poesnet model, using tensorflow.js. This required a fair amount of tweaking due to the overhead view.

Overhead view of people walking in a public space with data drawn on top

Center points

The next representation came from calculating the central point of each bounding box. We can also get a solid count on the number of in the feed at this stage.

Overhead view of people walking in a public space with data drawn on top

Group average

Finding the group average would now allow us to approximate average group X and group Y coordinates. At this point it was apparent that data smoothing would be required to deal with individuals entering/exiting the scene as well as general uncertainty in the model negatively impacting the animation.

Overhead view of people walking in a public space with data drawn on top

Final

For the final parameter, unity, a bounding box was drawn over the group‘s X and Y coordinates. Calculating the area of this bounding box allows us to approximate how close the individuals are to one another at any given moment.

We wanted to ensure that for the unity dimension, at a certain point of closeness (presumably 6 feet) there‘s no impact to prevent individuals standing closer than necessary.

Output

To experiment with the different parameters, I used mouse position to approximate group X and unity (measured on the Y axis). Clicking the mouse has the same effect as adding an additional person.