Usability research techniques for developer led growth
Hi there, welcome to my newsletter! I’m a product researcher in the dev tools space sharing my thoughts on how to use research to drive growth. If you enjoy this article, please subscribe.
Growth teams typically focus their efforts on acquisition and activation strategies for product led growth. Usability testing is a valuable method for identifying bottlenecks and optimizing for acquisition and activation. It involves task-based scenarios to observe user behavior and thought processes, providing insights into the ease of understanding and reaching the "aha moment" of your product.
Drawing from my experience in research and growth for developer tools, I’ll share techniques, example tasks, and key considerations to help you run effective usability tests.
When developers evaluate a new product, they typically seek answers to these critical questions:
What does your product do and what problem does it solve?
How much does this cost? If it’s not free, should I build or buy?
How easy is it to set up or to integrate with my application or stack?
1. What does your product do and what problem does it solve?
Developers want to know if your product will solve their pain point and save them time and effort. To assess this, it's essential to understand how easy or difficult it is to understand your product's purpose and relevance.
Tasks and/or questions:
Can you explore this page while thinking aloud and tell me who you think this is for and what this product does?
Now, we’ll go through an exercise where you’ll complete sentences to capture first impressions. Don’t think too hard and don’t worry about being right!
Key Considerations:
Observe where users click first, where they focus their attention, and what they skip.
If they go to docs, what code samples are they seeking? What language or framework? What do they plan to do with it?
If they stop somewhere on the page, what caught their eye?
Did they skip or gloss over something that is important to know?
Sentence completion is an effective method to draw out their understanding of your product. Write down exactly what they say, while sharing screen. Compare their answers with your understanding of the product, value proposition, and use cases.
Define your criteria for success in each task. Before you start testing, write out what you would consider ‘success’ or the right answer for each task.
Recruiting concerns: Consider if the user you’re testing with needs to be a backend, frontend or full stack developer. Do they need knowledge of a certain technology stack? Do you need to test with a buyer persona too? Do you need to test with someone who is familiar with the pain or problem that you are solving for?
Consider your task’s starting point. Do you need to start the test on your homepage or from search?
2. How much does it cost? If it’s not free, should I build or buy?
Aside from going to docs, almost all developers will check out your pricing page. They want to know if there's a free plan or free trial and if they have to pay, how much would it cost. Assessing whether the benefits outweigh the costs is crucial. They want to know the features they would get and the plan they need. The goal is to ensure that they can easily understand how pricing works, what plan would fit their needs and if pricing is a barrier to signup.
Tasks and/or questions:
Which plan would you choose, and why?
On a scale from 1 - 10 (1 being very difficult and 10 being very easy), how difficult or easy was it to find the right plan for you?
What do you expect to see or want to do after signing up?
Is there anything that would deter you from signing up?
Is there anything missing that would help you?
When would you consider building a solution in-house or using open source alternatives? When would a third-party service be preferable?
3. How easy is it to set up or to integrate with my application or stack?
Your aim is to observe developers set up your product. This could mean integrating with a sample app, their actual application or creating a POC.
Tasks and/or questions:
What does an ideal ‘hello world’ look like?
What do you expect to accomplish in 1 hour? What would be the expected end result?
Let's say you are ready to create a POC and test out this product. Can you show me how you would do that?
On a scale from 1-10, how difficult or easy was it to set up or integrate?
What were the most challenging aspects?
What do you think you accomplished at the end of this session?
Have you seen value from this product, and if so, at what point?
What would you like to do next?
What would increase your confidence in choosing this solution?
Key Considerations:
Determine if you need to provide a sample application or a demo environment.
Specify any prerequisites or setup requirements.
Decide whether users should start with docs or search.
Consider whether your idea of a hello world is the same as the user’s. Understand the user's expectations, their definition of the "aha moment," and their timeframe for creating a POC.
In cases where users struggle for a long time, offer assistance to maintain a balance between realistic problem-solving and progressing through the session. Keep track of challenges and help when it makes sense so that you may understand the whole setup journey.
To gather data for these three questions, consider dividing the research process into two sessions per participant. Additionally, please keep in mind that this guide is not an exhaustive list of what developers would assess for and may not be universally applicable to all developer products. There are different types of developers and you may market to different personas. Feel free to adapt and tailor these approaches to your specific context. Happy researching!