watsonx.data Trial Usability and First-use Concept Testing

Role: UX Researcher
Skills: Evaluative Research, Usability Testing, Findings Championing.

Due to the confidentiality of this project, certain details pertaining to the technologies tested and the solutions realized have been omitted.

Summary

Business-to-business software, especially products used by technical users, is increasingly adopting aspects of product-led growth (PLG). IBM's data lakehouse product, watsonx.data, adopted the model; however, the team wasn't seeing the conversion or even engagement with the product they'd anticipated. This succinct study uncovered 4 major recommendations and resulted in 20% more Trial plan sign-ups month-over-month (MoM), a 200% increase in accounts with daily rated usage (DRU), and contributed to the 320% increase in enterprise plan instances MoM.

Objectives

• Identify challenges that data engineers face when getting started with IBM’s data lakehouse
• Evaluate prototypes of proposed alternatives to the landing page

Process

Product analytics showed a lack of engagement with the trial experience that had been created for watsonx.data, IBM's data lakehouse product. My partner in product management hypothesized that engagement, which the product team had defined as number of queries executed, could be improved by landing users directly on the query page when they load into the trial. I disagreed. I felt this would cause confusion and a better solution existed; however, I needed data to make any claim for or against the decision. Therefore, I worked with my design partner to create a usability and concept test that we intended to run asynchronously. The unmoderated testing never took off due to delays and contextual business requirements, so the moderated pilot data were analyzed. Due to the low number of participants, qualitative data were prioritized (although quantitative data were also shared).

I conducted pilot tests with 2 internal target users, 2 external target users, and a usability expert. Originally, I intended to exclude the usability expert from the findings, but I decided to keep their data for specific tasks which did not require technical knowledge. Additionally, the work was presented out of a notes document, as my goal was to get a go/no-go for continuing the study based on the preliminary findings. Instead, the design and product management executives were ecstatic about the findings and began promoting work items to design and development based on my work. Talk about immediate impact!

Outcomes

Four recommendations were given as high-value fixes to the top findings of the study. These resulted in Github Issues (including a Sev 3 bug) and Aha! Tickets which were folded into the product. Additional recommendations relating to prior UX research work, product enhancements proposed by users, and lower-impact changes were provided in the appendix, as well as being submitted as tickets and comments on tickets in the proper channels.

While data presented were meant to be preliminary, the findings, including that our target users (data engineers) could not launch the product trial without assistance, were compelling enough to garner executive support. Other findings were clear usability heuristic violations, making it relatively easy to argue for changes to rectify the problems.

Concept testing data indicated that the homepage at the time did not adequately guide users, but the proposal to land users on the Query page was universally disliked. The decision was made to redesign the landing page instead of repurposing pages not intended to orient users.

Conclusions

As a result of this work, design changes led to 20% more Trial plan sign-ups MoM, a 200% increase in accounts with DRU, and contributed to the 320% increase in enterprise plan instances MoM.

Challenges

1. Finding stakeholders for ongoing work amid staffing changes
2. Communicating research to executives
3. Interpreting preliminary data