A/B Testing in Data Science Interviews by a Google Data Scientist | DataInterview

Hi Dan, for step 6, how do you get the confidence interval (3.4%, 5.4%)? Thank you.

In an A/B interview, is it common to need to calculate the Z-value or interview endpoints manually?

Excellent video!

Excellent content, I would appreciate just a slight improvement.
When you've assumed you want an alpha of 5% (in this case 2.5 per side) and you've also determined the beta. There is a different equation that gives you an output of how many people the experiment should be performed according to those assumptions. i don`t now wich one is better but from what i see is that yours doesn`t take into account the d of the experiment

more importantly, you can`t always use the sigma, it depends on several cases like the size of the test, independency of the samples and his unbiased estimator

the variance could be sigma OR S^2 OR sp OR SD

n>=(Sigma * (Z(1-a) + Z(1-Beta) / miu0 - miu1)^2

Hmmm, revenue per day per user. That seems much more variable than conversions per user sessions, which is binomial with bounded variance. Especially for clothing on an ecommerce site that isn't Amazon. Can you help me understand why you chose this as the primary success metric?

Dan - watching your videos helped me land a data scientist position at a FAANG company. So grateful for your knowledge and your ability to share it with a wide audience :)

your channel is GOLD! THANK YOU

awesome

The best video about A/B testing, definitely!

This is such a great video. This video is enough when you are preparing for analytics experimentation interviews

Welp, these videos tell me I am qualified to work at google in data science. Nevertheless, I don't see job openings in 2023. And I dropped out of school, so I've never been offered an interview despite my ability in statistics.

This is the best video I’ve seen about A/B testing. This is exactly a day to day DS work in tech companies. I very appreciate your sharing and looking forward for your new videos. Wishes you all the best!

@datainterview This is an amazing review of A/B testing in interviews! Quick question, if we have a minimal sample size value calculated, why do we also determine a duration of an experiment? I.e. if sample size is calculated to 500 users with searches per group, shouldn't the test duration be how long it takes a business to reach 500 users during the test period? And not an arbitrarily chosen time such as 1 to 2 weeks?

Can you explain same concept using jupyter notebook where you are following all the concepts that you talk in this video.?

This video made me subscribe to your channel. Extraordinarily detailed explanation of all of the things that go into experiment design.

The downside of an A/B test is it shows the non-optimal option too much. Look into Bandit Algorithms

I ni cash

Which statistical test did you use to get the p-value while interpreting the results

when you increase power (1-beta), you are actually increasing recall not precision

How do you get the p value 0.001 ?

Do you think that I need to get a course on data science first and then your course or I can jump straight into yours?

Hi Dan, Very informative video!

Would you have suggestions on how to select randomization units for B2B products like data bricks and slack, which have a many to 1 relationship between users and accounts?

This is amazing. Thank you. I will send you a Starbucks gift card if I get the job.

Great video. But I have a question. Isn't Step 5 Validity Check ought to be done before running the real experiment?

Very interesting ..well understood except the launch decision part .. the diagram with horizontal lines at different levels.

Wouldn‘t it violate the independent assumption if we use revenue per user per day for a/b testing? Since a high portion of low engaged users would just keep not buying anything.

great work. i love this course and it helped me alot.

Great content! Thank you!

Extremely helpful. Thank you for this step by guide to A/B Testing.

I am a product designer, and this is so great to learn. Thanks for explaining it in detail in a simple to understand way!

This is the best video about A/B testing in practice I've ever seen. Thank you for sharing

Amazing explaination and super concise, love it

Clear and helpful, thank you

Great content. Simple concise explanation.

I was asked a similar A/B testing question on a Data Science interview with LinkedIn. Had I seen this video a week prior, I would’ve made it to the 3rd round. Better late than never. Great content! I just subscribed.

Why do we use a two sided hypothesis test here? Can we not assume the new algorithm is at least the same as the old one and use the extra power?

This video literally made me subscribe to you channel Dan! I have gone through countless materials and research papers on experimentation and by this is one of the best summary of the steps! Great work and I am excited to see more videos from you(especially around pitfalls for multiple metrics and more)!

Excellent content as always Dan! As an expansion on the topic, could you consider doing a video on variation reduction techinques (e.g., CUPED, MLRATE) ?

Hi. Thanks for the Great video. At the moment I'm stuck with sample size, conversion rate during the test (let's say 2 weeks) and getting a histogram of the data. After 2 weeks I'll get one conversion rate. How can I get an histogram from only one data. "Inside" the CR there is a huge N (let's say 1000 users). I'm confused how can I extract the data from 1000 users to make a CR histogram. Could you help me out? Cheers

A very detailed explanation to get started on A/B Testing. Thank you so much.

Why is this a two sided hypothesis test instead of a straightforward one sided test?

this is so helpful

This video is awesome and really helpful if anyone wants to learn AB testing but doesn't have the time to finish the entire popular Udacity course. Thank you for making it so clear and interpretable!

please make a video on how the hypothesis is being tested and we get the p value/CI

??????? ?

Please upload more A/B testing videos. This topic is too confusing. Could you please upload a video explaining the power of hypothesis test and how to choose the sample size for an A/B test?

Thank you for the great content, really loved it. It will be great if you could have some videos around most commonly used statistical tests in depth and A/B testing pitfalls. Will keep watching！