Usability Test Case Study with Stytch: Part I
Recently, I wrote about usability testing as a way to identify friction points in developer products. The best way to the learn the benefits of usability testing is like how any developer learns a new product/framework: see an example and then try it yourself. :)
For this usability test, we will examine Stytch, an auth platform that offers SSO and other auth methods for B2B SaaS and consumer applications.
I invited Matt, an engineering manager at a B2B SaaS company with auth implementation experience to explore Stytch’s website. He recently evaluated Auth0 for a project, but has never looked into Stytch.
Disclaimer: The insights shared here stem from one participant. While they are helpful, they should be considered part of a broader study with more participants. The goal is to demonstrate how usability testing can yield actionable insights for developer products.
In the next section, I will describe how I approached this study.
Research Objective: Hypothetically, the team at Stytch is trying to understand how to drive signups. We will use usability testing to understand the new user experience and pinpoint friction. Referring to my previous article, we are trying to answer these questions:
How easy/difficult is it to understand the problem Stytch solves?
How clear is the pricing model and how easily can users determine costs?
How easy/difficult is it to integrate Stytch?
Although we didn’t have time for the integration, Matt explored what it would take to integrate Stytch. To answer Q3, I would run another usability session dedicated to it.
Here’s some context to set the stage:
Persona: Engineering manager at B2B SaaS startup with experience implementing auth and SSO (Single Sign-on). Proficient in .NET, TypeScript and React.
Usability Task: You are building an app and need to add authentication to it but don’t have time to build it in-house. You’ve heard about Stytch and are considering it for your project.
Your goal is to gather data through observation, listening and asking questions in order to answer the research questions outlined above. Refer to this article on questions you could ask along the way.
In the next section, I will detail main findings.
Let’s dive into the first research question: How easy/difficult is it to understand what Stytch is and the problem it addresses?
Matt understood that Stytch is an easy way to add authentication and SSO to apps. He rated the ease of understanding at 7/10 (1 being very difficult, 10 being very easy). But triangulating data is key here! Here are three ways that I assessed the clarity of messaging and documentation:
Rating scale: Matt’s 7/10 rating is good, but don’t just rely on this self-reported metric. :)
Sentence completion: After exploring the website and docs, I asked Matt to complete the sentence, “Stytch is….”
His response: “Stytch is… ‘Auth0 on steroids… it feels more modern, leaner, and seems built with developers in mind.”
This technique is incredibly insightful, revealing a perspective I hadn’t picked up before this exercise. He really liked the docs but hadn’t mentioned ‘modern’ or ‘lean’. This feedback is useful for the team managing the brand and website. If the marketing team expects different descriptors, it would be good to follow up and identify any gaps.
His description suggests he knows Stytch is an Auth0 alternative, but he didn’t clearly define what that means. To ensure he understands the product, I could have asked, 'What is Auth0?' and then, 'Complete the sentence: The three benefits of using Stytch are…’
I made sure Matt closed his tabs so he wasn’t looking at the website during the exercise. :)
Observe the user’s behavior, interactions and path: Matt focused on certain parts of the homepage and documentation that demonstrated that he understood what Stytch offers and was looking to see it would fit his requirements.
As a developer, he initially wanted to dive straight into the docs but spent some time on the homepage for the usability test. He was drawn to sections like the live code snippet, B2B SaaS features (SSO, MFA, etc.), SDKs, and the Stytch vs. Auth0 comparison page. His focus on these areas reflect what developers care about when evaluating a new product and suggests he was assessing whether Stytch meets his technical requirements around SSO and React support. His interest in the comparison page with Auth0 indicates that, given unique advantages, he might consider switching from Auth0.
See clip below:
In the docs, he first spent time on the integration diagram, which helped validate his expectations about how Stytch would work. He was then drawn to sections like ‘OAuth’, ‘How to use Stytch JWTs’, and ‘SDKs’. He also explored the Microsoft OAuth flow, which he had to set up in a recent project while evaluating Auth0. The ease with which he found this information suggests that the documentation is well-structured and organized. In general, he was interested in many areas of docs. If a developer dives deep into technical content to explore how the product aligns with their mental model and integrates into their existing frameworks and workflows, it may indicate a strong understanding of the product and its relevance to their needs.
See clip below:
By combining the rating score, sentence completion, and observing whether they navigate to the right places or are ready to setup, we can gauge both their understanding of the product and their interest in actually using it.
In the last section, I will cover additional insights and recommendations.
One idea to increase signups is to showcase the login UI above the fold on the homepage and make the full demo apps easier to find. Matt was excited about the product when he saw the login page with Stytch's live code snippet on the homepage, and again when he discovered this full demo app in the docs. Highlighting the login UI more prominently and improving access to the demo apps could increase signup conversion rates.
Another potential idea for increasing signups is targeting specific authentication methods in conjunction with popular languages or frameworks, such as landing pages around time-based OTP (authenticator apps) for React apps. Matt recalled he had to implement this type of auth recently. Since this is an enterprise feature in Auth0, there may be an opportunity to capture developers who are specifically looking for this functionality.
Build dedicated migration guides for different auth providers. When Matt landed on the homepage, he noticed the quote about how easy it is to migrate from Auth0 and saw the comparison page, but he struggled to find a migration guide. He expected to find a guide dedicated to Auth0 but after some time, found a migration guide that covers multiple auth providers. Linking to this migration guide from the comparison page would be helpful too.
A huge shoutout to Matt for being such a great participant in this usability test 🙏 . I would also recommend conducting this test with developers who are new to auth and SSO.
See Part II to learn about second research question on pricing.
Subscribe for updates, and thanks for reading!