What user tests do you run?
Hey everyone.
So I'm in the midst of launching Preferr, and I'm wondering what kind of user tests you all run in your UX/UI processes. So...
What kinds of user tests do you employ to validate a design or design decision?
What current tools do you use for user testing?
Is A/B testing something you'd like to do?
11 year product designer here. The most useful kind of testing you can do is a usability test. It’s by far the fastest way to uncover specific parts of your design that users struggle with.
A/B testing can be useful for websites/apps that get a lot of traffic and have a clear event they can track against. However, you have to have a significant amount of traffic for an A/B test to be useful.
Thanks Josh!
What would you consider to be significant traffic? I've been trying to pin down those numbers to set some baselines for statistical significance.
From my limited experience so far, statistical significance can be achieved with smaller sample sizes if the variants being tested have an obvious winner in terms of conversion rate. But of course that's not always the case and frequently a test is being run on variants that are quite similar.
The site is a great resource for exploring A/B testing and plugging in different numbers: https://abtestguide.com/abtestsize/
The big factors in calculating how long you need to run a test to get a significant result are:
The control conversion rate
What the conversion rate is before any tests are run
Expected improvement (%)
How much better we think the new version will do over the control (e.g. we think the new signup form will perform 15% better than the control)
Current traffic
How many visitors you get
If you have a crazy high conversion rate (>20%) then you can test with a lot less traffic. Same if you have a ton of traffic.
True. But it has to be very obvious winner. Often it's so obvious before the test that you might as well make the change and not even bother with testing.