Questions tagged [ab-testing]
Use it for question related to A/B testing
85
questions
14
votes
4
answers
4k
views
A/B testing - client wanted the test run 70/30
We recently ran an A/B test for a new sidebar in a checkout flow. New variant 70% of traffic, old 30% of traffic. We tried to get client to run at 50/50 but they were sure our version was an ...
- 141
0
votes
1
answer
62
views
How to not skew results from usability research?
I wanted to take my class design project up a notch by getting my designs in front of some users through an unmoderated usability test. Because of the platform I used for the test, I could only make ...
1
vote
2
answers
155
views
How can i interpret the observed difference in a/b test?
I conducted an a/b test but I was confused about how to interpret the result.
I got 0.5% difference between control and treatment groups and statistical significance for the difference (p <0.05).
...
- 11
1
vote
1
answer
144
views
How to determine Minimal Detectable Effect for AB Test with Click-through Rate as success metric?
I am conducting an AB test on my site. I want to decide what is considered a good MDE (Minimal Detectable Effect) when it comes to click-through rate. I can understand you can choose MDE when your ...
- 113
0
votes
2
answers
67
views
AB Test Dilemma [closed]
I am running an AB test on a site I launched. If I want to launch a new CTA button on my site and as a result, another section of my site performs worse, how do I account for this or how can I avoid ...
- 113
4
votes
1
answer
85
views
Can we use A/B testing for analyze new features?
Hello I'm quite newbie in UX, especially in A/B Testing methods.
In my opinion and after I read some of articles. A/B testing usually used for analyze some new call to action button, copywriting, ...
1
vote
2
answers
133
views
Best practices for technically executing AB testing on ecommerce site?
I am working on a site redesign for my company's ecommerce site. We sell access to upcoming and recorded webinars. As a part of this redesign, I have recommended a new product strategy in which ...
- 51
3
votes
2
answers
102
views
A/B testing usability with static user count
I'm new to A/B testing and I have few questions.
The situation
I would be testing an information system with no new users, so user count is more or less constant. In the system there is a big form ...
- 33
1
vote
1
answer
157
views
A/B Usability Testing Technique Format?
I'm currently preparing a usability test for a module interface I'm designing and I want to be able to structure the test to ensure the best results.
Currently, there are 3 versions of the interface, ...
- 369
1
vote
2
answers
92
views
Convention for users to opt out of A/B tests?
An associate was unable to follow a set of basic instructions I made for using a common website. It turned out my user was on some A/B test and was served a different UI. When I checked the same site ...
- 289
5
votes
0
answers
58
views
UI/Process Design For A/B Testing [closed]
For fixed horizon A/B tests, the sample size needs to be determined based on alpha, beta, effect size and variance prior to a test. Therefore, in the experiment UI setup page, we need to ask users to ...
- 151
0
votes
3
answers
110
views
Did not perform A/B Testing , how to define success of feature
The problem is regarding a educational matching platform. We match Learners together so they and the fellow learners can video call. Learners are entered into a ‘matching pool’ where they can see all ...
1
vote
2
answers
126
views
Test Hypothesis On Multivariate Tests?
I was taught when A/B testing every test should be based on a test hypothesis, e.g.
Control: Current Button (Blue)
Test: White Button
"We believe that changing the CTA colour (British English ...
2
votes
3
answers
255
views
Selecting the right user testing method for mobile app
We have a mobile app for different platforms. And now we want to improve the UX and UI. Due to the security policy, we cannot install analytic tools. I'm thinking of options we have to measure our ...
1
vote
1
answer
194
views
How important is it to avoid A/B test collisions?
Let's say you have 3 A/B tests as such
Control vs A
Control vs B
Control vs C
Unless you specifically work to ensure that tests do not collide (run on top of each other), you will end up with ...
- 151