bias-variance tradeoff

Bias-Variance Tradeoff

The Bias-Variance Tradeoff is a core concept in supervised learning and statistics describing the tension between two sources of prediction error. It is fundamental for diagnosing model performance and selecting appropriate model complexity.

Error Decomposition

Let $y = f(x) + \epsilon$ be the true relationship, with $\epsilon$ as irreducible noise ($E[\epsilon]=0$, Var$(\epsilon)=\sigma^2$). For estimator $\hat{f}$, the expected mean squared error decomposes as:

Bias

Bias measures systematic error from simplifying reality. High bias (underfitting) occurs when models like linear regression cannot capture complex patterns. Low bias means the model's assumptions fit the data well.

Variance

Variance measures prediction sensitivity to training data fluctuations. High variance (overfitting) occurs when flexible models like high-degree polynomial regression fit noise. Low variance yields stable predictions across datasets.

Irreducible Error

Data-inherent noise that no model can surpass—the fundamental error floor from unmeasured variables or measurement error.

The Tradeoff

Bias and variance are inversely related through model complexity:

• Simple models (e.g., linear regression, shallow decision trees): low variance, high bias.
• Complex models (e.g., deep neural networks): low bias, high variance.

Optimal complexity minimizes total error, producing a U-shaped test-error curve.

Diagnosis

Learning curves: high bias → both errors high and close; high variance → large gap between low training error and high validation error.

Remedies

• High bias: increase model complexity, add features, reduce regularization.
• High variance: add training data, simplify model, increase regularization (L1, L2), use Bagging (e.g., Random Forest).

Example: KNN

In KNN, $k$ controls the tradeoff: small $k$ → low bias, high variance; large $k$ → high bias, low variance. Cross-validation finds the optimal balance.