BESTBUY_VirtualTryOn_250225_ARCHIVE

TEAM
Sunny Lee - UX Lead
Devansh Garg - Product Management
Justin Morgan - iOS Lead
Andrew Bradnan - iOS Development
Harrison Shapley - Android Lead
Alexander Biemann - Android Development
Suresh Peddisetti - React Native
Mike Gourdin - Quality Assurance Lead
Aaron Wedin - Quality Assurance
Sudharsan Uthayasuriya - 3D Modeling Lead
Vairam Vignesh - 3D Modeling
Jonathan Bergeron - UX Research
Georgia Carlsson - UX Content
Dave Kover - UX Standards
Vidhya Subramanian - Accessibility

DURATION
6 months

CLIENT
Best Buy - Seattle Technology Development Center

TOOLS
Figma, UserTesting.com, MIRO, 3DStudioMax, VRay, Blender, Reality Composer, Photoshop, Illustrator, After Effects, Lottie, PowerBI Analytics, OpinionLab, and JIRA.

PLATFORMS
iOS: ARKit, SceneKit, and RealityKit
Android: ARCore and Sceneform

OVERVIEW

In late 2022, following the pandemic, Best Buy retail stores faced substantial losses due to theft, leading the company to implement protective enclosures for high-value products to reduce shrink. However, this measure restricted customers’ ability to try on smartwatches and glasses in-store, negatively impacting sales and increasing return rates as shoppers were unable to accurately assess sizing before purchase.

To mitigate the risks associated with high-value items on shelves while still enabling customers to virtually try on wearables, we were tasked with developing these augmented reality (AR) solutions to provide a comprehensive alternative to in-store fittings.

TARGET DEMOGRAPHIC

Through consultation with business intelligence expert Elyssa Mason, we determined that the success of smartwatches and glasses hinges on features, cost, and compatibility with users’ lifestyles. These devices appeal to a diverse audience, necessitating functionality across varying skin tones, eye colors, hand sizes, and lighting conditions. To meet company diversity, equity, and inclusion (DEI) standards, we partnered with Applause and Best Buy’s accessibility teams to refine detection capabilities and user interfaces.

WEARABLES

Smartwatches appeal to a wide audience because they seamlessly integrate into daily life, offering a blend of practicality, health benefits, and style. Whether it’s staying connected, tracking fitness, or simplifying tasks, smartwatches provide a convenient and efficient way to enhance everyday routines.

Companies like Gunnar introduced blue light glasses to reduce eye strain, while Meta and Bose made AR audio glasses popular among cyclists. Seeing the growing demand for smart wearables, we joined the trend to help customers find the right fit with ease.

Ray-Ban Meta glasses with open ear audio.

Gunnar Blue Light Reduction Amber Lenses

DESIGN CONSIDERATIONS

I conducted a competitive analysis to better understand the market landscape and customer expectations. Surprisingly, I found few players in the space. Notably, Warby Parker stood out as one of the only major competitors equipped to handle legal challenges, such as those posed by the Illinois Biometric Information Privacy Act (BIPA). This insight highlighted the unique positioning of Warby Parker in addressing both market demands and regulatory complexities.

Additionally, I assessed the Wanna Watch app, which showcased robust performance in virtual watch try-on experiences. In comparison, our experiments with Apple’s ML model revealed significant limitations. Specifically, the model was restricted to open-hand gestures, despite users naturally preferring a closed-fist posture for such interactions. These findings underscored critical areas for improvement and provided valuable direction for refining our strategy moving forward.

After evaluating key players in the watch try-on space, we spent several weeks attempting to develop our own hand-tracking ML model. However, upon further investigation of Wanna Watch, we discovered they’d partnered with SNAP Inc.. Recognizing the potential of leveraging their proven technology, we reached out to explore the possibility of a licensing agreement.

PERMISSIONS

Following the Target class action lawsuit on April 26, 2024, which accused the company of violating the Illinois Biometric Information Privacy Act (BIPA), our legal team prioritized mitigating similar risks for our organization. To address this, I worked with Erin Anderson-Kipp (content) to design and develop a biometric permissions message. The goal was to ensure compliance with privacy laws, improve transparency, and clearly communicate to users how their biometric data is collected, stored, and used. This initiative aimed to build trust with users and reduce the likelihood of legal issues related to biometric data handling.

FEATURE PRIORITIZATION

After a quick team discussion, we worked together to outline our strategy. Drawing on findings from the competitive analysis, we carefully assessed the potential influence of each feature. We also weighed whether users would likely expect specific capabilities in later versions, ensuring our development plan aligned with both customer needs and industry trends. This approach enabled us to prioritize features strategically and design a product that connected with our audience.

After consulting with our Data Scientist Rad Cyrus and UX Research Director Jonathan Bergeron, we moved forward using internal data that showed our approach could significantly reduce theft and alleviate stress for in-store workers.

MODELING

We kicked off the project by identifying our most popular glasses SKU and downloading an existing 3D model from our server. Once in hand, I collaborated with our lead engineers and our on-staff modelers to identify the optimal points of contact for the glasses and watch categories. After finalizing the pivot points, we created documentation and contacted vendors to ensure their 3D teams understood the updates.

We chose the bridge of the nose as the pivot point for all glasses models. Scaling was based on inter-pupillary distances (IPD), ensuring a custom fit for every user.

Due to the complexities of wrist tracking technology, extensive experimentation was required to determine whether this position could function as the pivot point. While it was ideal for scaling, achieving a stable positional lock without an open hand was uncertain. However, using SNAP’s ML model, we successfully secured a stable lock with a closed fist.

LOADING

While working with Luxottica, we found their 30MB 3D models were too large for most app users, causing delays, especially in low-bandwidth environments. Since our models are typically under 5MB, we optimized five files for the launch and trained their team in mesh optimization. I also developed a loading system to display progress, preventing users from staring at a streaming video of themselves without a 3D model superimposed.

Preview of loading glasses.

DYNAMIC MESSAGES

While the SNAP ML model was still being tested, I collaborated with UX Content Manager Georgia Carlson to design two messaging options. If the model proved ineffective, we would use the prompt “Keep open hand in view.”

The message surfaces when no hand or wrist is detected. Once detected, the message disappears, and a 3D watch is overlaid on the user’s wrist.

Internal Applause tests showed that when users’ faces weren’t detected, hiding the 3D glasses model (instead of adding an error message) was clear to them. Customer feedback later confirmed this. Users naturally understood that no model meant their face wasn’t detected, so we kept the design simple without extra alerts since resources at the time were already being dedicated to producing a loading model.

UI NORTH STAR

We initially designed a bottom toolbar with a menu drawer for browsing related SKUs. Users could navigate between watch series by swiping the drawer or using on-screen gestures. The feature also included and dimension tools to ensure accurate sizing.

For glasses, we adopted a similar user interface approach, differing primarily in the presentation of variations within the drawer and the display of dimensions to users.

MVP IOS

Ultimately, we reduced the features to the essential requirements, displaying product information and enabling SKU variation browsing through a carousel.

MVP ANDROID

For our Android developers, repurposing existing components from previous projects provided a quicker way to achieve functionality similar to the North Star design.

GLASSES

The Glasses Try-On feature developed in partnership with Meta and Luxottica was deployed on 17 October 2023, enabling real-time AR visualization to assist users in selecting optimally fitting eyewear styles. This innovation enhanced decision-making precision while providing a personalized, interactive interface.

WATCHES

On January 12 2024, the Watch Try-On feature was launched exclusively on Android. A functional iOS prototype was also developed and submitted to Apple for review but was not approved for release. This process underscored the political, technical, and platform-specific complexities of implementing advanced ML features across different ecosystems.

LESSONS LEARNED

In retrospect, a key mistake was inadequate communication with our Apple representative. We assumed that presenting a functional prototype would generate enthusiasm for the technology and its potential to connect users with their products. However, Apple was not interested in showcasing their products in 3D on our website or app.
Despite this, engagement with wearables and the virtual try-on feature remains strong, with placement rates significantly exceeding those for in-space product visualization.
We underestimated Apple’s share of the wearables market. Without approval, leadership chose not to advance beyond the MVP unless circumstances changed, leading to a halt in production.

CASE STUDIES