Why do we divide by n-1 to estimate the variance? A visual tour through Bessel correction

Correction: At 30:42 I write "X = Y". They're not equal, what I meant to say is "X and Y are identically distributed".

The variance is a measure of how spread out a distribution is. In order to estimate the variance, one takes a sample of n points from the distribution, and calculate the average square deviation from the mean.
However, this doesn't give a good estimate of the variance of the distribution. The best estimate, however, is obtained when dividing by n-1 instead of n.
WHY!?!?!?!?!?!?!?
In this video, we dig deeper into why the variance calculation should be divided by n-1 instead of by n. For this, we use an alternate definition of the variance, which doesn't use the mean in its calculation.

*[0:00] Introduction and Bessel's Correction*
- Introducing Bessel's Correction and why we divide by \( n-1 \) instead of \( n \) to estimate variance.

*[0:12] Introduction to Variance Calculation*
- Explaining the premise of calculating variance and introducing the concept of estimating variance using a sample instead of the entire population.

*[1:01] Definition of Variance*
- Defining variance as a measure of how much values deviate from the mean and outlining the basic steps of variance calculation.

*[1:52] Introduction to Bessel's Correction*
- Discussing why we divide by \( n-1 \) when calculating variance and introducing Bessel's Correction.

*[2:35] Challenges of Bessel's Correction*
- Sharing personal challenges in understanding the rationale behind Bessel's Correction and discussing my research process on the topic.

*[3:20] Alternative Definition of Variance*
- Presenting an alternative definition of variance to aid in understanding Bessel's Correction and expressing curiosity about its presence in the literature.

*[4:45] Quick Recap of Mean and Variance*
- Briefly revisiting the concepts of mean and variance, demonstrating how they are calculated with examples, and explaining how variance reflects different distributions.

*[7:05] Sample Mean and Variance Estimation*
- Explaining the challenges of estimating the mean and variance of a distribution using a sample and discussing why sample variance is not a good estimate.

*[8:49] Bessel's Correction and Why \( n-1 \) is Used*
- Explaining how Bessel's Correction provides a better estimate of variance and why we divide by \( n-1 \) instead of \( n \). Emphasizing the importance of making a correct variance estimate.

*[10:51] Why Better Estimation Matters?*
- Discussing why the original estimate is poor and why making a better estimate is crucial. Explaining the significance of sample mean as a good estimate.

*[13:02] Issues with Variance Estimation*
- Illustrating the problems with variance estimation and demonstrating with examples why using the correct mean is essential for accurate estimates. Explaining the accuracy of estimates made using \( n-1 \).

*[15:04] Introduction to Correcting the Estimate*
- Discussing the underestimated variance and the need for correction in estimation.

*[15:57] Adjusting the Variance Formula*
- Explaining the adjustment in the variance formula by changing the denominator from \( n \) to \( n - 1 \).

*[16:22] Calculation Illustration*
- Demonstrating the calculation process of variance with the adjusted formula using examples.

*[16:57] Better Estimate with Bessel's Correction*
- Discussing how the corrected estimate provides a more accurate variance estimation.

*[18:24] New Method for Variance Calculation*
- Introducing a new method for calculating variance without explicitly calculating the mean.

*[20:06] Understanding the Relation between Variance and Variance*
- Explaining the relationship between variance and variance, and how they are related mathematically.

*[21:52] Demonstrating a Bad Calculation*
- Illustrating a flawed method for calculating variance and explaining the need for correction.

*[23:37] The Role of Bessel's Correction*
- Explaining why removing unnecessary zeros in variance calculation leads to better estimates, equivalent to Bessel's Correction.

*[25:08] Summary of Estimation Methods*
- Summarizing the difference between the flawed and corrected estimation methods for variance.

*[26:02] Importance of Bessel's Correction*
- Emphasizing the significance of Bessel's Correction for accurate variance estimation, especially with smaller sample sizes.

*[30:19] Mathematical Proof of Variance Relationship*
- Providing two proofs of the relationship between variance and variance, highlighting their equivalence.

*[35:24] Acknowledgments and Conclusion*

Thanks @mkan543 for the summary!

Скачать видео