Usability Test Case Study with Stytch: Part II
This is a continuation of Stytch's usability test insights from Part I. If you haven’t read Part I, I recommend starting there for context, as it covers the first research question: how easy is it for developers to understand what the product does.
Apart from documentation, the pricing page is likely the most visited part of your website for developers. The goal of this study is to ensure clear and transparent pricing. Developers prefer a self-service experience that allows them to experiment with a free trial and quickly understand costs without needing to contact sales.
Disclaimer: The insights shared here stem from one participant. While they are helpful, they should be considered part of a broader study with more participants. The goal is to demonstrate how usability testing can yield actionable insights for developer products.
Let’s dive into the findings for second research question: How easy or difficult is it for developers to understand Stytch’s pricing?
Matt rated the ease of selecting a pricing plan at 5/10, with 1 being very difficult and 10 very easy. The main issue he encountered was the unclear distinction between B2B SaaS and consumer authentication. This confusion began even before he reached the pricing page—it started with the documentation. Early in the usability test, Matt chose 'Consumer Auth documentation without giving it much thought. Later, when reviewing pricing plans, he naturally focused on Consumer and overlooked B2B SaaS.
Researcher tip: It’s important not to correct participants in cases like this. I wanted to see whether Matt would realize on his own that he needed the B2B SaaS plan. When it became apparent he hadn’t, I eventually asked him what he thought the difference was between the two plans. Without my prompt, he might have continued setting up consumer auth, which could lead to issues if he needed features only available in B2B SaaS.
For context, B2B SaaS and consumer authentication refer to different types of applications and users. Orb, one of their customers, is an example of a B2B SaaS use case. They provide metered billing and can be used by large enterprises. Enterprise companies often need SAML SSO to ensure that their employees can securely access applications like Orb. Stytch’s SAML SSO feature allows Orb to sell more easily to enterprises by meeting this security requirement. In contrast, consumer auth is used by individual consumers—for example, Target could use Stytch’s auth for users to log into their personal Target accounts. In Matt’s case, as an engineering manager working for a small B2B SaaS company, he actually needed the B2B SaaS plan.
When I dug into why Matt immediately picked Consumer auth, I realized it came from a misconception about terminology. Matt associated ‘B2B SaaS’ with enterprise pricing and larger companies, while linking ‘Consumer’ to a cheaper developer plan. He didn’t realize that ‘B2B SaaS’ and ‘Consumer’ actually refer to the use case — specifically the application and its users. This highlights the importance of labels in information architecture. While we often focus on the structure and navigation, the specific labels significantly shape user behavior. Here’s how Matt explains his thought process:
Interestingly, Stytch seems aware of this potential confusion. As of September 4, 2024, they’ve updated their documentation to include a page that explains the difference between B2B SaaS and consumer authentication.
Because this is just one participant’s feedback, it is important to triangulate with other data sources to determine how widespread this issue is. I would look at:
Support tickets, Slack inquiries and other direct customer feedback channels: See if there are frequent questions about the difference between B2B SaaS and consumer auth.
Analytics and session recordings: Analyze user behavior to see if users frequently toggle between the two categories. Back and forth navigation could indicate a lack of clarity. See if there are more unique clicks on ‘Consumer’ but you actually get more B2B SaaS leads.
Usability testing: Conduct usability tests with scenarios that require participants to choose between B2B SaaS and consumer auth. Observe how often they select the correct category to gauge understanding.
Stytch clearly values developer experience as they’ve customized their docs and product for these two use cases. A small area of improvement would be ensuring new users can easily distinguish between the two categories and find the right product suite—if this confusion is indeed a significant enough issue.
Additional insights and recommendations:
Tooltip helpers: Adding a tooltip helper for M2M (Machine-to-Machine) would help clarify this feature for users unfamiliar with the term.
Pricing calculator ease of use: When Matt entered a low number, the pricing didn’t change, leaving him unsure whether his input affected the total costs. Providing immediate text or visual updates would have helped his understanding. He mentions this pain point in the clip above.
The main challenge with understanding Stytch’s pricing, ironically, isn’t about the specific pricing plans. It seems to originate from confusion around the two categories of authentication. If this issue is common, it's worth exploring ways to provide clearer explanations and guidance early in the user journey. A more deliberate flow—such as a wizard-style quiz asking key questions like “Who will be logging in?” and “What features do you need?”—could help guide developers toward the correct product suite. If anyone reading this has alternative suggestions, please share! 💚
This concludes the usability test findings for Stytch. Thanks for reading!