I realized that the team was constrained on developer resources and decided to create a Unity prototype to validate two key assumptions:
1. Users will notice the “Tilt phone up” message.
2. Users will click on “Tap to place” button.
I conducted a remote usability test with 20 users and found that the majority of the users were able to understand the new interaction and successfully place the 3D model on their room floor. I decided to further refine the concept.
We launched an A/B test comparing the new design with the old design to a small percentage of customers. While the test was running, we saw a significant drop in the number of customers placing the product. A detailed analysis showed that some customers did not tilt their phone up, and others did not "tap to place" the 3D model on the floor.
Both of these hypothesis had been tested before, but I realized that the participants in the user test were disproportionately more adept at technology than Amazon customer base. Great learning for future, but I had to move quickly to improve the design and launch the feature.
We came up with 3 improvements after looking at metrics and conducting another usability test with the launched app.
1. I proposed making the “Tilt phone up” very prominent by showing it in the middle of the screen. We had observed that significant percentage of customers were not tilting their phone up.
2. I proposed to keep the product visible once the customer had tilted their phone up and not make it disappear if they tilt their phone down again.
3. We also noticed that customers were first seeing the 3D models very far away from their feet. I proposed to reduce the distance of the minimum zone so that the customers can see the product sooner when they are tilting their phone up.
We relaunched another A/B test with all the changes above.
*Disclaimer: These are my personal views and do not represent the views of Amazon. The numbers are made up for confidentiality reasons.