Overfilling and Underfitting in Machine Learning

Overfitting occurs when a machine learning model is too complex and captures noise or random fluctuations in the training data, making it perform well on the training set but poorly on unseen data. Regularization techniques, more data, or simpler models can help mitigate overfitting.

The ipynb file for this tutorial is here.

https://github.com/niranjanmeegammana/ML-master/blob/main/04_TEC/MLC_overfitting_and_Underfitting_000.ipynb

import numpy as np

from sklearn.model_selection import train_test_split

from sklearn.tree import DecisionTreeRegressor

import matplotlib.pyplot as plt

# Generate some synthetic data

np.random.seed(42)

X = np.random.rand(100, 1)

y = 4 * (X - 0.5) ** 2 + 0.1 * np.random.randn(100, 1)

# Split the data into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train the decision tree with different depths

depths = [1, 3, 5, 10]

train_scores = []

test_scores = []

for depth in depths:

    model = DecisionTreeRegressor(max_depth=depth, random_state=42)

    model.fit(X_train, y_train)

    train_score = model.score(X_train, y_train)

    test_score = model.score(X_test, y_test)

    train_scores.append(train_score)

    test_scores.append(test_score)

# Plot the results

plt.figure(figsize=(8, 6))

plt.plot(depths, train_scores, label='Training Score')

plt.plot(depths, test_scores, label='Testing Score')

plt.xlabel('Depth of Decision Tree')

plt.ylabel('R^2 Score')

plt.title('Overfitting Example with Decision Tree Regressor')

plt.legend()

plt.show()

Here, we generated a set of  synthetic data and used a decision tree classifier with different depths (1, 3, 5, and 10). As the depth increases, the model becomes more complex, and the training accuracy increases while the testing accuracy decreases. This indicates overfitting, as the model is memorizing the training data rather than generalizing to unseen data.

# Train the decision tree with depth 5

depth = 5

model = DecisionTreeRegressor(max_depth=depth, random_state=42)

model.fit(X_train, y_train)

# Create a meshgrid to plot the decision boundary

x_min, x_max = X.min() - 0.1, X.max() + 0.1

xx = np.linspace(x_min, x_max, 100).reshape(-1, 1)

# Predict target values for the meshgrid points

yy = model.predict(xx)

# Plot the decision boundary and the training data

plt.figure(figsize=(8, 6))

plt.scatter(X_train, y_train, label='Training Data')

plt.scatter(X_test, y_test, label='Testing Data', marker='x')

plt.plot(xx, yy, c='r', label='Decision Boundary')

plt.xlabel('Input Feature (X)')

plt.ylabel('Target Value (y)')

plt.title('Decision Boundary for Decision Tree Regressor')

plt.legend()

plt.show()

the red line represents the decision boundary generated by the decision tree regressor. As the depth increases, the model becomes more complex, and you may observe overfitting, where the decision boundary tries to fit the training data too closely, resulting in poor generalization to unseen data points (represented by 'x' markers).

Underfitting occurs when a machine learning model is too simple to capture the underlying patterns in the data. As a result, it performs poorly on both the training and testing data. This typically happens when the model is not complex enough or when there's insufficient data for the model to learn meaningful relationships.

import numpy as np

import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LinearRegression

# Generate some synthetic data

np.random.seed(42)

X = np.random.rand(100, 1)

y = 3 * X + 2 + 0.1 * np.random.randn(100, 1)

# Split the data into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train a linear regression model

model = LinearRegression()

model.fit(X_train, y_train)

# Predict on training and testing data

y_train_pred = model.predict(X_train)

y_test_pred = model.predict(X_test)

# Plot the data and the linear regression line

plt.figure(figsize=(8, 6))

plt.scatter(X_train, y_train, label='Training Data')

plt.scatter(X_test, y_test, label='Testing Data', marker='x')

plt.plot(X_train, y_train_pred, c='r', label='Linear Regression Line')

plt.xlabel('Input Feature (X)')

plt.ylabel('Target Value (y)')

plt.title('Underfitting Example with Linear Regression')

plt.legend()

plt.show()