Architecture of Convolutional Neural Network (CNN)

Convolutional Neural Network (CNN)  is a deep learning model designed for processing structured grid data, such as images. The data is represented as a mathematical grid. The CNN model consists of layers such as convolutional, pooling, and fully connected layers. CNNs capture hierarchical features in images to perform  tasks such as image classification, object detection, and image generation.

A Convolutional Neural Network (CNN) typically consists of the following types of layers:

An example code for this tutorial 

1. Convolutional Layer: 

 This layer performs convolutions on the input image using learnable filters or kernels. These filters help to detect various features like edges, corners, and textures in the image.

2. Activation Layer:

After the convolution operation, an activation function like ReLU (Rectified Linear Unit) is often applied element-wise to introduce non-linearity to the network.

The calculations in convolutional layer 

3. Pooling Layer:

Pooling layers reduce the spatial dimensions of the data, which helps in reducing the computational complexity and controlling overfitting. Common pooling methods include max pooling and average pooling.

4. Fully Connected Layer:

This layer connects every neuron to every neuron in the previous and subsequent layers. It's used to make final predictions or decisions based on the features learned by earlier layers.

5. Flatten Layer:

 Before connecting to the fully connected layer, the 2D output of the convolutional and pooling layers needs to be flattened into a 1D vector.

6. Normalization Layer:

These layers are used to standardize the inputs to the network, making the optimization process more stable.

7. Dropout Layer:

 Dropout is a regularization technique used to prevent overfitting. During training, randomly selected neurons are ignored, reducing the reliance on any particular neuron.

8. Output Layer:

The final layer of the network that produces the desired output, which could be a classification label, regression value, or any other relevant prediction.

These layers are stacked together to create a deep architecture capable of learning hierarchical features from input data. The arrangement and combination of these layers can vary depending on the specific CNN architecture and the task at hand.

Soft-Max Layer

Soft-max is an activation layer normally applied to the last layer of network that acts as a classifier. Classification of given input into distinct classes takes place at this layer. The soft max function is used to map the non-normalized output of a network to a probability distribution.

• The output from last layer of fully connected layer is directed to soft max layer, which converts it into probabilities.
• Here soft-max assigns decimal probabilities to each class in a multi-class problem, these probabilities sum equals 1.0.
• This allows the output to be interpreted directly as a probability.

For binary classification problem, logistic function is used and for multi-classification soft-max is used.

The functionality of  Convolutional Layer in a CNN is a mathematical operation known as convolution, which is a fundamental operation in image processing and computer vision. It is used to extract features from input images. The convolutional layer functions are :

1. Filters (Kernels):

These learnable filters also known as kernels, are a small matrix that slides or convolves over the input image.

2. Local Receptive Field:

While  the filter slides over the input image, it looks at a small local region of the input called the receptive field. The size of this receptive field is determined by the dimensions of the filter.

3. Element-Wise Multiplication and Summation:

At each position of the receptive field, the filter's values are element-wise multiplied with the corresponding pixel values in the input image. The results of these multiplications are summed up to produce a single value.

4. Feature Map Generation:

As the filter slides across the entire input image, a new matrix , called a feature map is generated. This feature map captures the presence of certain features or patterns in the input image.

5. Shared Weights:

 In CNNs, the same filter is used across different regions of the input image. This is known as weight sharing, and it allows the network to detect the same features in different parts of the image.

6. Multiple Filters:

 The convolutional layer typically contains multiple filters, each responsible for detecting different features in the input image. This allows the network to learn a variety of features at different levels of abstraction.

7. Strides:

The filter can move across the input image with a certain step size called the stride. A larger stride results in smaller output feature maps, as it reduces the number of positions the filter visits.

The convolutional layer's ability to slide filters over the input image and capture local patterns makes it well-suited for tasks like edge detection, texture recognition, and more complex feature extraction in images. These learned features are then passed on to subsequent layers for further processing and higher-level feature extraction.

Activation layer

An Activation Layer in a Convolutional Neural Networks (CNNs) is used to  introduce non-linearity to the network's computation. This functionality is crucial because many real-world data patterns are inherently non-linear.

Activation functions are applied element-wise to the outputs of the previous layer's neurons. 

The Activation Layer functions are:

1. Linear Combination:

Before passing through the Activation Layer, the inputs to each neuron are computed as a weighted sum of the outputs from the neurons in the previous layer. This linear combination represents the "activation" of the neuron.

2. Application of Activation Function:

The activation function is then applied to the activation value. This function transforms the linear activation into a non-linear output. Without this non-linearity, the network would behave like a single-layer perceptron, which is limited in its learning capacity.

3. Non-Linearity: Activation functions introduce non-linearity by mapping the input values to a new range. This enables the network to learn complex relationships between features in the data.

Common Activation Functions used in neural networks, including CNNs, include:

- ReLU (Rectified Linear Unit):

 ReLU(x) = max(0, x). It replaces negative values with zero and leaves positive values unchanged.

- Sigmoid:

Sigmoid(x) = 1 / (1 + e^(-x)). It squeezes the input into the range (0, 1).

- tanh (Hyperbolic Tangent):

Tanh(x) = (e^(x) - e^(-x)) / (e^(x) + e^(-x)). It maps input to the range (-1, 1).

- Softmax:

 Primarily used in the output layer for multi-class classification. It converts a vector of real values into a probability distribution.

tanh:

The hyperbolic tangent function, abbreviated as "tanh," is similar to the sigmoid function but maps its input to a range between -1 and 1, making it zero-centered. The tanh function is defined as:

tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))

The output of the tanh function lies between -1 and 1, which means it can produce both positive and negative values.

The tanh activation function is often used in cases where the output of a neuron needs to be centered around zero and have a stronger response to negative values compared to the sigmoid function. 

Furthermore, the tanh function helps mitigating the vanishing gradient problem compared to sigmoid, it may still suffer from vanishing gradients for very large or very small inputs.

The choice of activation functions depends on the nature of the problem, the architecture of the network, and the avoidance of potential issues like vanishing gradients. The introduction of non-linearity through the Activation Layer is essential for the network's ability to learn complex patterns and relationships in the data.

Pooling Layer : 

The primary function of the pooling layer in a CNN is to reduce the spatial dimensions of the input data while retaining important features. Pooling helps in managing computational complexity, reducing overfitting, and creating a more abstract representation of the input data. The two most common types of pooling layers are max pooling and average pooling.

1. Max Pooling:

   Max pooling operates on each local region of the input independently. In each region, it selects the maximum value and discards the rest. This helps in capturing the most prominent feature within that region.

2. Average Pooling:

   Average pooling, on the other hand, computes the average value of the elements within each local region of the input. It creates a more smoothed representation of the data.

Pooling layers introduce translation invariance to some extent, meaning that small shifts in the input data won't drastically change the pooled output. This is particularly useful in computer vision tasks where the exact position of features in an image might vary.

In both max pooling and average pooling, we define the size of the pooling window and a stride. The size of the pooling window determines the reduction in spatial dimensions, while the stride determines how much the window shifts.

Fully Connected Layer:

This layer plays a crucial role in transforming the features learned from earlier layers into final predictions or decisions. Here,  each neuron is connected to every neuron in the previous and subsequent layers.

The convolutional and pooling layers extract hierarchical features from the input data, typically images. The purpose of the fully connected layers is to learn complex relationships between these features and make high-level decisions based on them.

Key characteristics of fully connected layers are:

1. Connections: Each neuron in a fully connected layer is connected to every neuron in the previous layer. This means that the output of every neuron is influenced by all the input features.

2. Weights and Biases: Each connection between neurons has associated weights and biases, which are learned during the training process to optimize the network's performance on the given task.

3. Non-Linearity:Fully connected layers often include an activation function (such as ReLU, sigmoid, or tanh) after the weighted sum of inputs and biases. This introduces non-linearity to the network, allowing it to learn more complex relationships in the data.

4. Output: The final layer of fully connected neurons produces the network's predictions or decisions. For example, in image classification, the output layer might have neurons corresponding to different classes, with each neuron representing the likelihood of the input image belonging to a specific class.

Fully connected layers are powerful tools for capturing intricate patterns and relationships in the data. However, as the number of neurons in these layers increases, the model's complexity grows, which can lead to overfitting if not properly regularized. Dropout, batch normalization, and other techniques are often employed to mitigate this issue.

A fully detailed video series on CNN

Niranjan Meegammana 

Cyber Security and ML Researcher

Sri Lanka Institute of Information Technology

Shilpa Sayura Foundation