Gradient Descent optimization algorithm in Deep Learning

Gradient Descent is an optimization algorithm used to minimize a function by iteratively adjusting the model's parameters in the direction of the steepest descent (negative gradient) of the function with respect to those parameters. It is widely used in machine learning and deep learning for training models.



1. Initialization: 

Start with an initial guess for the model's parameters (weights and biases). This can be random or based on some prior knowledge.


2. Calculate the Gradient: 

Compute the gradient of the loss function with respect to each parameter. The gradient represents the direction and magnitude of the steepest increase in the loss.


3. Update Parameters: 

Adjust the model's parameters by moving in the opposite direction of the gradient. This update is done iteratively, and the magnitude of the update is controlled by a hyperparameter called the learning rate (α). The formula for parameter update typically looks like: parameter = parameter - (learning_rate * gradient).


4. Repeat: Continue steps 2 and 3 for a fixed number of iterations (epochs) or until a convergence criterion is met (e.g., when the gradient becomes very small).


There are different variants of Gradient Descent, including:


- Batch Gradient Descent:

Computes the gradient using the entire dataset at each iteration. It can be slow for large datasets.


- Stochastic Gradient Descent (SGD): 

Computes the gradient using only one randomly chosen data point at each iteration. It can have high variance but converges faster and can escape local minima.


- Mini-Batch Gradient Descent: 

Computes the gradient using a small random subset (mini-batch) of the dataset at each iteration. This is the most commonly used variant as it combines some of the advantages of both Batch and SGD.


Gradient Descent is a fundamental optimization technique used to train a wide range of machine learning models, including linear regression, logistic regression, neural networks, and more. Properly tuning the learning rate and monitoring convergence is crucial for its success in training models effectively. Additionally, there are advanced optimization algorithms, like Adam, RMSprop, and Adagrad, which are variants of Gradient Descent that adapt the learning rate during training for improved convergence.


Gradient Descent :  Linear Regression example 


import numpy as np

import matplotlib.pyplot as plt


# Generate synthetic data

np.random.seed(0)

X = 2 * np.random.rand(100, 1)

y = 4 + 3 * X + np.random.rand(100, 1)


# Add a bias term (intercept) to X

X_b = np.c_[np.ones((100, 1)), X]


# Initialize model parameters

theta = np.random.randn(2, 1)


# Set hyperparameters

learning_rate = 0.1

n_iterations = 1000


# Gradient Descent

for iteration in range(n_iterations):

    gradients = -2 / 100 * X_b.T.dot(y - X_b.dot(theta))

    theta -= learning_rate * gradients


# Print the learned parameters (theta)

print("Learned Parameters:")

print("Intercept (theta_0):", theta[0][0])

print("Slope (theta_1):", theta[1][0])


# Plot the data and regression line

plt.scatter(X, y)

plt.xlabel("X")

plt.ylabel("y")

plt.title("Gradient Descent Linear Regression")

plt.plot(X, X_b.dot(theta), color='red')

plt.show()




Gradient Descent  logistic regression 



import numpy as np

import matplotlib.pyplot as plt


# Generate synthetic data

np.random.seed(0)

X = np.random.rand(100, 2)  # 100 samples with 2 features

y = (X[:, 0] + X[:, 1] > 1).astype(int)  # Binary classification


# Add a bias term to X

X_bias = np.c_[np.ones((X.shape[0], 1)), X]


# Logistic sigmoid function

def sigmoid(z):

    return 1 / (1 + np.exp(-z))


# Initialize model parameters (weights)

theta = np.random.rand(3)


# Set hyperparameters

learning_rate = 0.01

epochs = 1000


# Gradient Descent

for epoch in range(epochs):

    # Calculate the predicted probabilities

    logits = np.dot(X_bias, theta)

    y_pred = sigmoid(logits)

    

    # Calculate the gradient of the loss function (log-likelihood) w.r.t. parameters

    gradient = np.dot(X_bias.T, (y - y_pred))

    

    # Update parameters using Gradient Descent

    theta += learning_rate * gradient


# Now, theta contains the learned parameters for the logistic regression model

print("Learned Parameters (theta):", theta)


# Plot the decision boundary

plt.scatter(X[:, 0], X[:, 1], c=y)

x_boundary = np.linspace(0, 1.2, 100)

y_boundary = (-theta[0] - theta[1] * x_boundary) / theta[2]

plt.plot(x_boundary, y_boundary, color='red')

plt.xlabel("Feature 1")

plt.ylabel("Feature 2")

plt.title("Logistic Regression Decision Boundary")

plt.show()



Gradient Descent  : Artificial Nueral Network


Gradient Descent is a fundamental optimization algorithm used for training Artificial Neural Networks (ANNs). ANNs consist of multiple layers of interconnected neurons (nodes), and the goal of training is to find the optimal set of weights and biases that minimize a specific loss function. Gradient Descent plays a critical role in this process. Here's how Gradient Descent is applied to training ANNs:


In this code : 

We compile the model using Stochastic Gradient Descent ('sgd') as the optimizer and binary cross-entropy as the loss function.


Then, we train the model using the model.fit method, specifying the number of epochs and batch size. This is where Gradient Descent is applied iteratively to update the model's parameters.


import numpy as np

import matplotlib.pyplot as plt

from sklearn.datasets import make_moons


# Generate moon-shaped data

X, y = make_moons(n_samples=1000, noise=0.1)


# Visualize the data

plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)

plt.title('Generated Moon-Shaped Data')

plt.show()





import tensorflow as tf

from tensorflow import keras


# Generate some synthetic data for demonstration

# Split the data into train and test sets


split_ratio = 0.8

split_index = int(len(X) * split_ratio)


x_train, x_test = X[:split_index], X[split_index:]

y_train, y_test = y[:split_index], y[split_index:]



# Build a simple feedforward neural network

model = keras.Sequential([

    keras.layers.Dense(32, activation='relu', input_dim=2),  # Input layer with ReLU activation

    keras.layers.Dense(16, activation='relu'),             # Hidden layer with ReLU activation

    keras.layers.Dense(1, activation='sigmoid')            # Output layer with Sigmoid activation (for binary classification)

])


# Compile the model

model.compile(optimizer='sgd',  # Stochastic Gradient Descent

              loss='binary_crossentropy',  # Binary cross-entropy loss for binary classification

              metrics=['accuracy'])


# Train the model using Batch Gradient Descent

history = model.fit(x_train, y_train, epochs=100, batch_size=32, validation_data=(x_test, y_test))


# Evaluate the model

test_loss, test_accuracy = model.evaluate(x_test, y_test)

print(f"Test Loss: {test_loss}, Test Accuracy: {test_accuracy}")



Visualization


import matplotlib.pyplot as plt


# Plot training & validation loss values

plt.plot(history.history['loss'])

plt.plot(history.history['val_loss'])

plt.title('Model loss')

plt.ylabel('Loss')

plt.xlabel('Epoch')

plt.legend(['Train', 'Validation'], loc='upper right')

plt.show()



# Plot training & validation accuracy values

plt.plot(history.history['accuracy'])

plt.plot(history.history['val_accuracy'])

plt.title('Model accuracy')

plt.ylabel('Accuracy')

plt.xlabel('Epoch')

plt.legend(['Train', 'Validation'], loc='lower right')

plt.show()




Niranjan Meegammana

Cyber Security and ML Researcher

Sri Lanka Institute of Information Technology

Shilpa Sayura Foundation


Comments

Post a Comment

Popular posts from this blog

AI-Typical Phrases to Avoid in Academic Writing

Human Academic Replacements of AI Phrases 1. Filler Opening Phrases (Delete or Rewrite) Avoid (AI-like) Human Academic Alternative In today’s fast-paced world (Delete entirely) It is important to note that (Delete) It should be noted that (Delete) It is worth mentioning that (Delete) There is no doubt that Evidence shows that It can be seen that This shows that One can observe that This indicates that This paper aims to This paper examines 2. Overused AI Transition Phrases Avoid Replace With Furthermore Also Moreover In addition That being said However On the other hand In contrast At the same time Meanwhile In conclusion In summary Overall speaking Overall From a broader perspective More generally 3. AI-Style Explanatory Phrases Avoid Replace With To put it simply Simply At its core Essentially This means that As a result In other words That is This suggests that This indicates that This demonstrates that This shows that 4. Formulaic Emphasis Patterns Avoid Pattern Human Rewrite Not o...
Read more

Tips for Humanizing AI Text

Image
To reduce the AI likelihood in your document and make it more human-like, follow these strategies: 1. Personalize the Writing Style Add personal insights, opinions, or unique perspectives rather than just factual descriptions. Use more conversational or varied sentence structures to break AI-like patterns. 2. Introduce Variability in Sentence Length AI-generated text often has a consistent structure . Try using shorter and longer sentences alternately to mimic natural human writing. Use rhetorical questions or emphasize key points with italicized or bold text (where applicable). 3. Avoid Overuse of Structured Formatting AI-generated documents tend to overuse structured elements (e.g., numbered lists, bullet points, tables). Reduce the number of tables where possible and explain key points in flowing paragraphs instead. 4. Add Real-World Examples and Case Studies Instead of generic descriptions, incorporate real-world applications, anecdotes, or case studies to...
Read more

Overfilling and Underfitting in Machine Learning

Image
Overfitting occurs when a machine learning model is too complex and captures noise or random fluctuations in the training data, making it perform well on the training set but poorly on unseen data. Regularization techniques, more data, or simpler models can help mitigate overfitting. The ipynb file for this tutorial is here . https://github.com/niranjanmeegammana/ML-master/blob/main/04_TEC/MLC_overfitting_and_Underfitting_000.ipynb import numpy as np from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeRegressor import matplotlib.pyplot as plt # Generate some synthetic data np.random.seed(42) X = np.random.rand(100, 1) y = 4 * (X - 0.5) ** 2 + 0.1 * np.random.randn(100, 1) # Split the data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Train the decision tree with different depths depths = [1, 3, 5, 10] train_scores = [] test_scores = [] for depth in depths:     mo...
Read more