Outlier detection and removal: z score, standard deviation | Feature engineering tutorial python # 3

Thanks very much for your simple and clear code.

can you provide mock interview?

why we choose height column ??why dont we chose weight column???

Thanks!

How standard deviations is selected as 3 and zscalar 3 too?
Please someone explain

Excellent explanation in every topics, it really helps me alot for my data science career.. thanks

how can you apply this rule when you have about 10 features? Do you do them one by one?

Great Video. Thx!!

You can also use seaborn to plot the bell curve. It's much easier than matplotlib method.
seaborn.histplot(data=df.height, kde=True)
kde is the kernal density estimate line

Everything is good when you are applying Z_score for searching outliers which are either positive or negative outliers. If both positive and negative values are present together then it does not work..!!
data = [1, 2, 2, 2, 3, 1, 1,-19, 2, 2, 2, 3, 1, 1, 2,19,25]
try with this simple dataset.
with IQR method you can detect -19,19,25 all three
but with Z_score it is not working.
I don't know the reason. If you know Sir then let us know.

A little suggestion to make it simpler. In Z-Score method I can calculate its absolute value through np.abs and I can only write < 3 in my condition for the new dataframe.
In addition, to visualize the curve it is better to use sns.histplot with kde=True

I think thanks is not enough for your teaching skills ! really amazing 👌eye opened me about outliers !!

BRUH.... why would you remove one column .... this just ruins the propose

Wah... extra-ordinary explanation sir. Thank you...

you are simply amazing , yr simple explanation helping a lot , thanks a trillion

hello sir, can we learn personally from you? and how can we contact you

Great tutorial, thanks for using readily available sample CSV as well. ☑☑

Nice effort

Your videos are easy to understand. Thanks so much!

Really amazing lecture sir,i increasing interest on Data science sir

Your tutorial is so clear. Well done!

How do we determine the Z-Score range for Skewed data? Do I use the same range on either side (like -3 to 3) or can I use different values like -1 to 3 (for left skewed data) after looking at the histogram plot?

Thanks in advance!

Does this work only if the feature is normally distributed? Most of the features in real world data are not normally distributed.

woww! what a simple and easy to understand tutorial. Love it. Thank you sir.

Thanks!

You know python, but you dont know much about statistics in identifying the outliers in normal distributed data.

Hello! Your lesson is very helpful for me. Can you just say how can I find outliers using multiple parameters? Like I want to find the outliers using all the column of data together that I have. What should I do??
Thank you in advance.

I have a question kindly answer. Suppose we have 20 column and from all 2 column we are removing outliers, then we are excluding small amount of data from each column, i.e. all together we are loosing huge data. Is this a correct way to handle outliers ?

Removing outlier is good option of replacing outliers with other value is good option ?

Thank you!

hey. why cant we use 'StandardScaler' and delete all outliers ?

Thank you so much sir your way of teaching is so clear and easily understandable

How to decide 3 as a threshold value to calculate zscore values? you have considered ex: zscore >3

man!! I was struggling with how to use statistics in EDA. I knew std, mean n all but couldn't use them in the EDA flow. u just cleared my confusion!!!! u won't believe how long I have been struggling with this.. thank god I found this video.. u r a great teacher.. I had the tools but couldn't use them. u just taught me how to use it..

Sir reviewer has asked me this question I don't know how to address it, can you please guide me "Use some statistical significant test such as T-test or ANOVA to prove you validate the proposed diagnostic model on patients and quality improvements of your method". I have two datasets. Dataset 1 was used to train the model and dataset 2 was used to validate the trained model. I have trained the ML model deployed it and Validated it on new data and presented the results. Actually, I have understood the question. Shall I apply the statistical test between the performance metrics of trained model results and validation results? Please help me, sir.

🙌🙌🙌

how to remove outlier from dataframe which has categorical as well as continuous data, as by percentile technique I am getting NaN value in categorical columns

Sir thank you for this topic, it's very useful. Can you make more outlier remove tutorial such as use unsupervised learning for outlier detection.

Very nice

How can we apply this to multiple columns?
Is there any short way or we have to do it manually for every column?

It is a really beneficial and useful video on this topic, thank you!

Very well explained sir!!
Worth watching

Hi Sir,
How do I decide Z score values, does it depend on my data or is it always -3 to +3?

Question, say you have a df of drink consumption and if you don't want to eliminate the outliers but instead replace them with NaN and keep the zero values of the dataframe, what would you do? Thanks

Tks for the very detailed explanation sir...

Don't worry about the histogram plotting code, one simple alternative is -

import seaborn as sns
sns.distplot(df['XYZ'])

Great Greaaaaat and a fulll too Greaaattttt explanation man. Loved it.