Bias vs Variance Tradeoff

Bias is, assumptions made by a model to make the target function easier to learn; therefore :

  • LOW BIAS : refers to less assumptions made about the form of the target function.
  • HIGH BIAS : refers to high assumptions made about the form of the target function.

Low-bias machine learning algorithms include: Decision Trees, k-Nearest Neighbors and SVMs (Support Vector Machines).
High-bias machine learning algorithms include: Linear Regression, Linear Discriminant Analysis and Logistic Regression.

Ont the other hand, Variance is the estimate value of change of Target function if different training data was used, therefore:

  • Low Variance: Suggests small changes to the estimate of the target function with changes to the training dataset.
  • High Variance: Suggests large changes to the estimate of the target function with changes to the training dataset.

Low-variance machine learning algorithms include: Linear Regression, Linear Discriminant Analysis and Logistic Regression.
High-variance machine learning algorithms include: Decision Trees, k-Nearest Neighbors and Support Vector Machines.

Overfitting

  • Overfitting occurs when a statistical model or machine learning algorithm captures the noise of the data.
  • Intuitively, overfitting occurs when the model or the algorithm fits the data too well.
  • Specifically, overfitting occurs if the model or algorithm shows low bias but high variance.
  • Overfitting is often a result of an excessively complicated model,

Underfitting

  • Underfitting occurs when a statistical model or machine learning algorithm cannot capture the underlying trend of the data.
  •  underfitting occurs when the model or the algorithm does not fit the data well enough.
  • underfitting occurs if the model or algorithm shows low variance but high bias.
  • Underfitting is often a result of an excessively simple model.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s