WIP [ITP Blog]

Experiments in AR - Week 7 [Midterm]

October 22, 2019

The first idea I had for my Experiments in Augmented Reality midterm project was a sports jersey with AR target markers on it so that fans could customize the player names on the jersey in an augmented space. But as I dug deeper into this idea, I wondered why this blank canvas jersey had to be limited to sports or even clothing. Why not allow anyone to write custom messages in an augmented space? An application that allows messages to be shared in a hybrid public-private manner in physical spaces could even be useful in an activist setting, such as Movers and Shakers’ AR based Monument Project or a digital version of China’s Democracy Wall in the late 1970’s. The application would also be an interesting platform to explore the “public-ness” of augmented reality space. An AR topic increasingly discussed is who controls it and what barriers to entry exist for AR based content. Working through this midterm project made me think how the artwork in Apple and the New Museum’s [AR]T Walk is gatekeeped by the custom AR application installed on the Apple Store’s iPhones that are lent out to existing Apple customer visitors.


Another interesting factor I wanted this project to explore is how the physical space itself affects the messages left behind. How does the content, language, and tone change between AR messages left at the ITP lobby versus city hall? I was inspired by the geolocation decisions made by the artists on the Apple [AR]T Walk, especially John Giorno’s drifting poem placement in the sky above a lake in Central Park.

Where Thoughts Go is an example of exploring how people share their thoughts and explore others in a “public” VR space, but the physicality is still dictated by the designer and not the existing physical environment.

Public Requirements

In order to make this AR project as public and accessible as possible, I aimed to make it web-based. This would remove the app download boundary and hopefully allow for the broadest device support. Currently the W3C is working on a WebXR API that would define a standard for developing VR and AR browser-based experiences that utilize internal device capabilities (e.g. Apple’s ARKit, Android’s ARCore), but the API is still in flux. For example, any WebXR experiences built on the current API requires a special browser on iOS devices because the technology is still new. So WebXR was not the most easily accessible, plug-and-play option for this project.

Another Web AR alternative that’s popular for commercial use is 8th Wall, but it didn’t feel right to rely on proprietary AR software for a project dealing with public space (and they charge money). So I decided to use the popular open-source JavaScript based AR.js library. Jerome Etienne, AR.js’ original maintainer, mentioned in a JSConf.Asia talk how he started this library to bring AR experiences to the broadest audience, a sentiment that’s inline with my midterm project.

Building Public Message

The prototype I built for this midterm is a Ruby on Rails web application called Public Message. I decided to use Rails because I needed a backend infrastructure with a database to create, store, and show saved messages and it’s the web framework I’m most familiar with. The first thing I built in that prototype was a static test webpage based on the AR.js basic example:

testing AR.js

Then I modeled out how I would create, store and show messages in Public Message:


For this initial prototype, I created two anchors — one for the ITP lobby and one for classroom 413 in 370 Jay Street. Each anchor has a unique URL and when that URL is scanned via QR code the user can see the associated message with that anchor. If this is the first time an anchor has been visited, then the user can write the first Post. If another person has already posted a message at this anchor, then the current user can submit a response that will be displayed on the anchor for future visitors. I used Three.JS as my 3D rendering library within AR.js, so text messages were written to an HTML canvas, converted into a Three.JS Texture, the texture applied as a material with a geometric object (I used PlaneGeometry) to a Three.JS mesh, and then the mesh attached to the rendered AR.js scene. I relied on this Texture-From-Canvas Three.js example for rendering text in a 3D space.

Here is a video demo of rendering a message associated with an Anchor record using AR.js:


Here is a video demo that includes the button and forms to add new posts and responses to an existing anchor:

Future Thoughts

I plan on continuing this project as part of my Socially Engaged Art and Digital Practice course, so there are some changes and additional features I’d like to add to Public Message in future iterations. Some of these were insightful suggestions made by my peers during the midterm presentation:

  • How should the entire message thread of posts and comments be displayed? Is it layered in AR or scrollable? Is there a link to read more (similar to how we have expandable comment sections on the web)?
  • Build out the feature for anyone to create new anchors
  • Is there a way to make a stronger connection between the message left and the location (e.g. color, audio, etc)?
  • Experiment with displaying starter prompts at new anchors to direct the conversation in a certain direction
  • Change the target marker — the Hiro image is the default marker and people thought “Hiro” had some significance to the project
  • Fix text wrapping in the canvas
  • Add a map feature to see where all the anchors exist in the world

Adrian Bautista

A perpetual work in progress blog documentating my NYU ITP projects. Words are my own.