ARMaps - Explorations

I set out to explore the advantages and disadvantages of exploring 3D data using Augmented Reality and hand gestures.

Design goals

I set up some constraints around future hardware limitations and interaction methods:
1. An AR headset with a decent enough field of view (FOV > 65)
2. Fully articulated hand tracking as the input mechanism

Hardware

I built a custom video passthrough AR headset using Oculus Rift, ZED Mini camera, and Leap Motion device for hand tracking.

Input modality
Bi-manual vs single handed

There are several limitations with the bi-manual hand interaction method:

1. A user cannot interact with the system for longer periods of time because of hand fatigue.
2. Users need to occupy both of their hands to interact with the system, leading to accessibility issues and/or situational disabilities.

User has to pinch any one of their hands in order to activate rotate, zoom, and pan operations. Benefits of single handed interaction:

1. It is relatively less energy consuming to interact with the system over longer periods of time.
2. The user needs only one of their hands to interact with the system resulting in a more inclusive system.

Getting the details right
Interaction surface orientation

Exploring a 3D representation of an urban part is done best on a horizontal surface as humans are used to seeing the ground as horizontal. Sci-fi movies like "Inception" break this rule to evoke wonder and communicate dream worlds. On the other hand when looking at a text-heavy map, humans tend to prefer reading/exploring it while holding it in their hands at an angle.

Taking cue from this I decided to automatically change the orientation of the map depending on the level of zoom. So the user can easily explore the 3D model of a city in the horizontal orientation of the map and when the zoom level becomes low such that the map becomes text rich, the map is gradually tilted to a 30 degrees orientation from the vertical for maximum readability.

The map also automatically rotates to orient north towards the up direction as human beings intuitively associate the north direction with up while reading a map. Rotation functionality is disabled when the map is in vertical mode.

Visual affordances

The system recognized the pinch action as the user's index finger and thumb finger coming closer than 3.5 cm. I added a visual signal to communicate to the the user that the pinch action had been recognized by the system. I also added a trail behind the 3D cursor to add visual history of the finger for the user. This was helpful, especially when frame rates were low, to reorient the user about how much they had moved their hands and in which direction.

3D cursor

The system recognized the pinch action as the user's index finger and thumb finger coming closer than 3.5 cm. I added a visual signal to communicate to the the user that the pinch action had been recognized by the system. I also added a trail behind the 3D cursor to add visual history of the finger for the user. This was helpful, especially when frame rates were low, to reorient the user about how much they had moved their hands and in which direction.

Edge of the map

The map needed a boundary in order for the users to get familiar with the interactive surface and increase familiarity with the map. A large map with constantly changing boundaries and edge rotation quickly leads to becoming a distraction from the main task: exploration of the world.

Iteration
Zoom levels

It is hard to make accurate assumptions about hand travel when it is moving away/close from/to your eyes. Try holding your index finger right in front of your eyes at a comfortable distance. Now try to move the index finger 1 cm away from your eye. You will have a hard time gauging the distance that your finger has moved, while this is reasonably easy to do if you move your finger in a vertical or horizontal direction. 

The way that the level of zoom is controlled often lead to the user moving their hands away/close to their eyes. This made it very hard for the user to gauge how much zooming in/out they had already done.

Due to the network limitations of loading map tiles at different zoom levels and intensive rendering tasks on the GPU, there will be an occasional frame drop during user. If the user was in the middle of a zoom operation when the frame drop happened, they would lose context of how far they had zoomed in the map.

To solve these problems I decided to include a visual cue in the form of a square that scaled up or down depending on the amount of change in the zoom level. The square gradually animated back to the original boundaries of the map at the end of the zoom operation/pinch. 

This visual had an unexpected effect of communicating the boundaries of the map at the start of the zoom during the zoom operation. I further included a translucent square at the original boundaries of the map to communicate that the boundaries of the map have not changed and the blue square will come back to the original boundary at the end of the zoom.

Where is north?

We have all seen that person using Google Maps on there phone trying to walk somewhere, reverse their direction and then walk again trying to figure out where they are supposed to walk. Not knowing which direction you are facing on a map can be one of the most frustrating experiences. I decided to add a compass in the interface as people are familiar with a compass.

The compass when touched reorients the map to north, taking cues from mobile based map interfaces. The compass animates to raise a little bit from its resting position to represent the state that it can be touched to reorient the map to north.