ARt - Augmented Reality Touch

    • Immersion Corp
    • 2018
    • LA Times Festival of Books

AR Painting & Paint Mixing in HMD

with additional mobile "Magic Window"

The goal of ARt was multifaceted: to research new interaction methodologies for 3D spatial interface, show the value of a new small form factor actuation wearable (in this case a ring), and to create a fun experience that could be shown at tradeshows on both mobile and PC platforms.

The user in the HMD holds a palette in their left hand and can interact with it or draw with their right. The ring contains a button for selection and both the ring and palette are tracked with 6 degrees of freedom. In addition to drawing with digital paint, the user can also mix new colors of paint on the palette or draw pathways of varying textures. These pathways allow the mobile viewers to spawn a small avatar to control around the pathway, like a racetrack, and can also change their viewpoint from an AR hologram experience to a VR first person perspective of their avatar. The haptics felt on ring are determined by interaction states and the velocity of painting while the mobile haptics are determined by the HMD wearer's brush strokes in observation mode or by the speed of the mobile user's avatar when in avatar mode.

My role was to propose and create interaction design flows for the HMD and mobile experiences, suggesting hardware for the pallete and HMD, sound design, haptic design, and to work with the team to supervise the contractors we employed to aid with the visual design of the holograms.

This highlight reel is from LA Times' Festival of Books in 2018.

Palette innovations:

The 3D UI concepts proposed and implemented were designed with the intention of supporting both single and two-handed interactions. The two-handed interaction is referred to as the “quick action” menu. It can be accessed by tilting the palette toward the user, as if peeking over it. This exposes the quick action menu that requires the primary drawing hand to access and enables the user to undo the last action, clear their palette, or clear the entire scene. The destructive nature of these interactions is why I elected to require both hands.

The single hand interaction, dubbed the “layer cake,” can be accessed by tilting the palette to the left until it is perpendicular to the ground. The layer cake will continue to track positions with the palette until the user enters the selection state by holding down a button, freezing the menu in place and allowing the palette to pass through the hologram. If the selection state is exited while the palette is within an option, it is selected. In this demo, the layer cake is used for saving and loading scenes, but was originally conceptualized for selecting between different brush types (another vertical layer cake was to be used for saving and loading). These single hand interactions, in conjunction with haptic feedback, seem to successfully afford the ability to develop muscle memory which was the motivating factor in this design.

Insights from users:

It can be challenging for users unfamiliar with 3D interfaces to become used to moving in the forward axis (likely due to the conditioning from 2D screens). In the future, I aim to add affordances for depth as an extension of the hover system. Currently, the hover on the paint blobs will rotate the paint and play audio with haptics, but augmented reality still presents challenges (which will also be remedied when this supports hand occlusion rendering). However, I believe that raycast-like strands which animate to pull the user towards the interaction may enable depth guidance for the user.

Testing earlier versions of the HMD palette interactions in Unity.
Simulated startup and paint selection examples as experienced in the HMD.
Simulated paint example as experienced in the HMD.
Simulating the quick action menu as experienced in the HMD.
Simulating the gesture to enter the layer cake menu as experienced in the HMD.
palette prototyping

I experimented heavily with the palette shape and size to determine optimal placement and scale for holographic and palette objects.

Selected Work

More detail and projects available upon request.

ARt - Augmented Reality Touch

ARt - Augmented Reality Touch

Shown at LA Times NewStory 2018
Haptic Studio 2

Haptic Studio 2

Quick generation of haptic textures

Copyright © 2021  William Rubenstein. All rights reserved.