Amazon - Improving how to place items

I joined the team in mid-2019 as the second Product Designer directly reporting to the Head of Product. The design team was very small and I immediately took on the responsibility of redesigning the core experience of shopping on Amazon using Augmented Reality.

Over the next 3 months, I partnered with a product manager to define the product roadmap and successfully executed on it while growing the team. I created new processes for design, engineering and QA to ship high quality products and managed up by getting buy-in on product strategy.

A result of this effort was a multi-million dollar improvement to the Amazon business and a solid product foundation for future iterations.

My Role

Product Strategy - I closely partnered with a product manager, conducting research and analysis to create a product roadmap and prioritize areas of improvements. I advocated for defining customer focussed KPIs to keep the team aligned during the project.

Design and User testing - I designed, prototyped and user tested 3D interactions for both iOS and Android platforms. I performed design QA during the development process and kept the team focussed on shipping a high quality product.

Leadership - I managed up getting buy-in on designs and scoping decisions from senior leadership. I created concepts and prototypes to align the stakeholders on the product roadmap.

Setting the vision
Building the product roadmap

I conducted a comprehensive UI audit and a heuristic analysis to identify usability problems and prioritize key areas of UX improvements: product placement flow, onboarding, and cohesive visual design.

We also performed a funnel analysis and noticed that there was a big drop in the number of customers placing a product on the floor. Since we had 3 months to design, build, test and ship the feature, we decided to focus on product placement flow. We aligned on 2 customer pain points to address:

1. Customers place the 3D model on their feet, resulting in confusion as the 3D model fully blocks the camera view. (Validated by metrics analysis)
2. Customers do not understand that the 3D model is on the floor and thus do not trust the scale of the 3D model. (Based on existing usability studies)
Impact on the business
Define success

We conducted a metrics analysis and found that majority of our customers were placing the 3D model very close to their feets. We decided that we had to increase the placement distance from the customer's feet and track it. We also expected that this will lead to increased interaction with the 3D model as customers will be more likely to interact with the 3D model if they can view it appropriately.

Defining KPIs

I collaborated with the PM in defining the KPIs as follows: Revenue, % of successful product placement and successful interaction with the product.

We also considered increasing the time spent in the AR experience as a KPI, but decided against it as we wanted the customer to either purchase the product or decide it does not go well with their room as soon as possible.

Design, prototype and iterate

I came up with two ideas to address the customer problems. The first one was to detect walls and furniture in a customer's room and auto-magically place the 3D model in an empty space in the room. The Lead engineer mentioned that the wall detection and object detection technology was not mature enough yet to implement this..

The second idea was to ask the customers to "Tilt their phone up" as soon as we had detected the floor. We also decided that we should show the model only when it is fully visible in the small FOV of the mobile device. This was based on the assumption that viewing the 3D model in full view will lead to better comprehension of fit and aesthetics.

I explored different visuals to represent the floor detected by ARCore and ARKit. I defined a 3D cursor that shows up in the center of the mobile screen placed on the floor. The 3D cursor is the size of the bottom face of the 3D model and acts as if it is revealing hidden dots on the floor when the 3D model is dragged around.

I realized that at larger distances this 3D cursor will be oblique to the camera and will not be visible to the customer. I investigated creating a folding effect that makes the dots rise up just outside the product boundary. This solved the problem but was found to be too technologically complex to implement and was removed from the scope of the release.

Testing and de-risking

I realized that the team was constrained on developer resources and decided to create a Unity prototype to validate two key assumptions:

1. Users will notice the “Tilt phone up” message.
2. Users will click on “Tap to place” button.

I conducted a remote usability test with 20 users and found that the majority of the users were able to understand the new interaction and successfully place the 3D model on their room floor. I decided to further refine the concept.

Refining the concept

Once I realized that the concept was working sufficiently well, I set about identifying the edge cases and generalizing the concept to work for products of all sizes from Amazon echo to a large sofa, or a vertical lamp.

I discovered that the size of the 3D model was unknown before it was downloaded and thus we could not show the 3D cursor at the appropriate size. I came up with a constant size loading cursor that shows up before the product has been downloaded on customer device. Once the product is downloaded the 3D cursor scales to become the approximate size of the product base.

I realized that different sized objects should become visible at different distances from the customer's feet because of the difference in their screen space area. Bigger objects will need to have a larger minimum radius to be reasonably visible in the small viewport of a mobile screen. So I defined the minimum radius beyond which the product shows as a function of the product size.

First A/B test

We launched an A/B test comparing the new design with the old design to a small percentage of customers. While the test was running, we saw a significant drop in the number of customers placing the product. A detailed analysis showed that some customers did not tilt their phone up, and others did not "tap to place" the 3D model on the floor.

Both of these hypothesis had been tested before, but I realized that the participants in the user test were disproportionately more adept at technology than Amazon customer base. Great learning for future, but I had to move quickly to improve the design and launch the feature.

Fast Iteration

We came up with 3 improvements after looking at metrics and conducting another usability test with the launched app.

1. I proposed making the “Tilt phone up” very prominent by showing it in the middle of the screen. We had observed that significant percentage of customers were not tilting their phone up.

2. I proposed to keep the product visible once the customer had tilted their phone up and not make it disappear if they tilt their phone down again.

3. We also noticed that customers were first seeing the 3D models very far away from their feet. I proposed to reduce the distance of the minimum zone so that the customers can see the product sooner when they are tilting their phone up.

We relaunched another A/B test with all the changes above.

We achieved a multi-million dollar increase in Amazon product sales.
We increased the average distance where the customer placed the 3D model.

We realized we had made too many changes at once in the release, that made it difficult to identify design decisions that moved the business results. As a result it became hard for us to iterate on the feature to improve the right metrics. We incorporated new processes to ship the minimum number of changes in a release and split releases if necessary.

I also realized that getting good results in a usability test on a prototype does not mean that every customer will be successful. This is especially the case with AR as customers are experiencing AR for the first time. I proposed to incorporate extra time for prototyping and iterating in our planning for future projects.

*Disclaimer: These are my personal views and do not represent the views of Amazon. The numbers are made up for confidentiality reasons.