A/B testing - Statistical significance for beginners

What alpha level do you recommend?

I have 2 tests i ran on the same "change" one AB test with a broad goal criteria (ex all devices) and another AB test with anarrow criteria (ex. tablet traffic) - the broad had 10.000 conversions and a slightly lower CVR. (70% significance) The narrow had a bit higher CVR. (95% significance) but also much fewer samples. What would you then conclude is winning? 1000 (95% signif.) vs 10.000 (70% signif.) ? It seems like the formula is not bulletproof when very low sample sizes can get very high significance just because they show more extreme results.

Very nice and clear explanation on how to use the tool. However, they have a sample size calculator that for the same experiment you're doing, says you need way bigger samples... What should we do?

A/B test idea: Just change the word in the title from 'dummies' to 'beginners.