| Creative Coding

Smell-O-Vision

Cover Image for Smell-O-Vision

Background

This project has been a collaborative effort between myself and M Dougherty. I was incredibly excited to participate in their vision: to create an interactive, sensory experience with visuals and scents.

Planning

After some planning and feasibility experiments we decided to aim for an interactive environment that utilized camera based position detection to find the number of individuals in a space, the center point between them, and some other metrics discussed here. These datapoints would power a visualization using the P5 library and trigger scent diffusers around the room.

My primary role was to develop the machine learning integration, linking it with a live webcam and tuning it for performance in low light. M tackled hacking the diffusers so that they could be triggered from a single arduino. We both collaborated on the artistic direction for the visuals.

Execution

Visuals

My previous post discussed the basic setup of the Posenet machine learning model. As it turns out the most useful data points for our v1 were the number of individuals (poses), and their average X and Y coordinates in the room. The white circles in this photo represent individual poses; the purple circle is the center point for the group.

Top down view of individuals in a lobby. A purple blob is in the center of the photo and smaller white circles are on each individual.

Feeling pretty good about the model’s ability to handle stock footage, my next mission was to experiment to find possible visualizations that could use this data. Our initial attempts at visuals used the HSB colormode and had a pretty brutal color palette no matter the saturation. The results of my experimentation were maybe interesting but they weren’t much better I think.

In the end, we knew that we wanted four different color scenes in each corner and that they should be at least vaguely red, blue, green and yellow. After a few days of experimenting, our solution was pretty simple. Whenever the groups central point crossed into a new zone, we draw the blobs in standard RGB color mode using only the channel that corresponds with that zone.

User testing with my lovely wife Raven

Performance

After settling on a visual aesthetic, the performance of our visualization (as measured with frame rate) was starting to chug and lag. I found that the most impactful improvement was significantly shrinking the P5 canvas size, then scaling it up in the browser with CSS. This ended up creating some weird misalignments between the generated Posenet positions and the newly distorted canvas. However, after correlating the video, canvas, and Posenet input resolutions, it effectively solved all of our animation lag problems.

Installation Issues

At this point, we felt comfortable beginning to move our work into our installation space (a conference room at the NYU ITP campus). When using a proper webcam instead of the my Macbook’s, we ran into quite a bit of trouble finding a place for the camera due to a change in aspect ratio. We wanted the lights to be as dim as possible to enhance the visibility of the projection, but this lower light caused Posenet to struggle. In addition, we had a hard time finding a position for the camera that could see the room in its entirety. As a result, we couldn’t get any blobs to appear.

Installation Solutions

After another series of video and canvas size tweaks, we got some poses, but they would disappear eratically due to the lighting. After some experimentation, our current solution is two-fold: adjust our code to permit poorer quality poses and using post processing software to increase the exposure. It’s a super fine line between under and over detection (more blobs than people), but I’m pretty happy with the balance we’ve tuned it to at the moment.

Finally, for the camera positioning, we simply couldn’t manage to find a position that included the four quadrants of the room; I wished out loud for one of those iPhone fish eye lenses so we could tape it to the webcam. A few minutes later – in a moment of pure magic – M comes into the room brandishing a tragically unused gift from their mom: an iPhone fisheye lens. To the great delight of M’s mother, it worked like a charm.

Me setting up the FishEye lens in the corner of the room

Takeaway

This project was a tremendous opportunity for me to experiment with a variety of topics from machine learning, to animation, performance, and visualization. Historically, I have a difficult time delegating and trusting partners. This project, with it’s size, scope and deadline, would have been impossible task alone. However, with our shared creative vision and work ethic, M and I were able to create something really special. Getting to experience the gestalt of a truly collaborative, creative endeavor has been an incredibly rewarding lesson I’ll carry with me.

| Creative Coding

Three.js – Week Two

Cover Image for Three.js – Week Two

Overview

This week I wanted to learn more about camera movement and effects in three.js. I also wanted to experiment with bringing in outside data into the animation. M and I are working on a project to visualize group dynamics in a small room, so I started with laying data on top of a video feed using regular canvas elements. The ”levers” we were looking for in order to control the visualization were person count, average group X and Y coordinates, and a proximity or unity factor describing how close the audience was in relation to one another.

After I was able to successfully get these paramaters from a video feed, I started experimenting what an output would look like using three.js.

Input

Sourcing the data from a video feed occurred in a series of measurements that build off of one another. Throughout the process, I drew directly onto a canvas with regular canvas drawing tools to ensure I was describing the data correctly.

Posenet

The first representation of the group comes the Poesnet model, using tensorflow.js. This required a fair amount of tweaking due to the overhead view.

Overhead view of people walking in a public space with data drawn on top

Center points

The next representation came from calculating the central point of each bounding box. We can also get a solid count on the number of in the feed at this stage.

Overhead view of people walking in a public space with data drawn on top

Group average

Finding the group average would now allow us to approximate average group X and group Y coordinates. At this point it was apparent that data smoothing would be required to deal with individuals entering/exiting the scene as well as general uncertainty in the model negatively impacting the animation.

Overhead view of people walking in a public space with data drawn on top

Final

For the final parameter, unity, a bounding box was drawn over the group‘s X and Y coordinates. Calculating the area of this bounding box allows us to approximate how close the individuals are to one another at any given moment.

We wanted to ensure that for the unity dimension, at a certain point of closeness (presumably 6 feet) there‘s no impact to prevent individuals standing closer than necessary.

Output

To experiment with the different parameters, I used mouse position to approximate group X and unity (measured on the Y axis). Clicking the mouse has the same effect as adding an additional person.

| Interface Lab

I/O Project: Smart Koozie

Cover Image for I/O Project: Smart Koozie

Pitch

Come up with a simple application using digital and/or analog input and output to a microcontroller. Make a device that allows a person to control light, sound, or movement using the components you’ve learned about (e.g. LEDs, speakers, servomotors).

Background

In college, my roomate Alex and I came up with the idea for a smart glass that could be used by waitstaff to track customers‘ drink levels. Way back in our first week of Interface Lab we pitched some fantasy projects, and I picked the smart glass. After some really productive group brainstorming, we refined the idea into more of a koozie situation. I decided to run with this idea for our I/O project.

Fantasy device brainstorming miro

My plan was to use an force sensing resistor (FSR) to detect the weight of the drink (minus the glass itself). This data would be processed, fed into a browser for display in real time. Through this process I learned a lot about analog sensors, data smoothing, serial communication, as well as 3D modeling and printing.

Modeling and Printing

Over the past couple of weeks I‘ve been slightly obsessed with the 3D printers in the lab. I figured this would be a great opportunity to test my skills and learn more about 3D printing and modeling in the process. My idea had a few pieces that needed to fit together, so when choosing 3D modeling software, I optimized for features that emphasized precision and ended up with Autodesk‘s Fusion 360.

Final models captured in Fusion 360

The FSR sits in the middle plate which is dropped into the hollow sleeve. There‘s a portal at the bottom of the sleeve for the sensor‘s headers to poke out. The piece on the right serves to concentrate the weight of the glass and drink onto the FSR in the middle.

Printed elements from model

I was really happy with the end result. Although this was relatively simple to print, the modeling got pretty intricate. Spending a lot of time in the modeling software paid off when it came time for assembly.

Software

Arduino

The software on the arduino side is relatively simple, however I learned quite a bit about data smoothing of analog sensors in arduino. Offloading this data processing from the browser to the arduino seemed prudent for scaling the dashboard beyond a single sensor reading.

Printed elements from model

Initially, I attempted to implement a threshold based strategy, only updating if a certain threshold was crossed. However, given the weight of the glass and the sensor size/quality, the reading was particularly jittery. I ended up with a much cleaner and smooth result simply taking the average of the last 100 readings and printing that over time. After that, I mapped the sensor value (minus force due to the glass itself) to a range from zero to one hundred.

Browser

I used the serialport library along with standard web sockets to read the value from the arduino and feed it into a little dashboard built with React. Once I got a clean 0-100 value from the Arduino, I built a Glass component that took the percentage as a prop and rendered a glass svg. The liquid in the glass uses CSS transforms to scale according to the liquid percent.

If a value comes along that is far below the 0-100 range, that means the glass itself has been removed from the Koozie. This info was also passed along to the glass so the SVG could be dimmed.

Result

I‘m really happy with how this project came together. I was able to break down the requirements into discrete problems that felt manageable. There‘s a lot of new concepts here for me: Arduino, 3D modeling, serial communication, to name a few. It was really nice being able to incorporate existing strengths in web tech to complement and build off these new skills. This project pushed my horizons quite a bit, it‘s certainly pushed the boundaries of what I‘m capable of creating. I feel like I‘ve gained so many new tools to draw on for creating interactive experiences.