Architectures of CNNs (LeNet, AlexNet, VGG, etc.) - Deep Learning Tutorial

Convolutional Neural Networks (CNNs) have revolutionized the field of computer vision and become the backbone of various image-related tasks. Several influential CNN architectures, such as LeNet, AlexNet, and VGG, have significantly impacted the development of deep learning and computer vision research. In this tutorial, we will delve into these iconic CNN architectures, understand their key features, and explore code examples in Python using TensorFlow.

1. LeNet

LeNet, introduced by Yann LeCun in 1998, is one of the earliest CNN architectures. It consists of seven layers, including three convolutional layers, two subsampling (pooling) layers, and two fully connected layers. LeNet was initially designed for handwritten digit recognition.

Key Features of LeNet:

Simple architecture with small filters (e.g., 5x5) and pooling windows (e.g., 2x2).
Uses the sigmoid activation function in the hidden layers and a softmax function in the output layer.
Employs gradient-based optimization techniques for training, such as stochastic gradient descent (SGD).

Here's an example of building LeNet using TensorFlow's Keras library:


    import tensorflow as tf
    from tensorflow.keras import layers, models

    # Create a Sequential model
    model = models.Sequential()

    # Add Convolutional layers
    model.add(layers.Conv2D(6, kernel_size=(5, 5), activation='sigmoid', input_shape=(image_height, image_width, num_channels)))
    model.add(layers.MaxPooling2D(pool_size=(2, 2)))
    model.add(layers.Conv2D(16, kernel_size=(5, 5), activation='sigmoid'))
    model.add(layers.MaxPooling2D(pool_size=(2, 2)))

    # Flatten the feature maps
    model.add(layers.Flatten())

    # Add Fully Connected layers
    model.add(layers.Dense(120, activation='sigmoid'))
    model.add(layers.Dense(84, activation='sigmoid'))
    model.add(layers.Dense(num_classes, activation='softmax'))

2. AlexNet

AlexNet, proposed by Alex Krizhevsky et al. in 2012, is a deep CNN architecture that gained immense popularity after winning the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012. It marked a turning point in the field of computer vision, demonstrating the power of deep CNNs in image classification tasks.

Key Features of AlexNet:

Deeper architecture with eight layers, including five convolutional layers and three fully connected layers.
Uses the rectified linear unit (ReLU) activation function, which helps mitigate the vanishing gradient problem.
Employs data augmentation and dropout techniques to prevent overfitting.
Utilizes parallel processing with two GPUs to accelerate training.

Here's an example of building AlexNet using TensorFlow's Keras library:


    import tensorflow as tf
    from tensorflow.keras import layers, models

    # Create a Sequential model
    model = models.Sequential()

    # Add Convolutional layers
    model.add(layers.Conv2D(96, kernel_size=(11, 11), strides=4, activation='relu', input_shape=(image_height, image_width, num_channels)))
    model.add(layers.MaxPooling2D(pool_size=(3, 3), strides=2))
    model.add(layers.Conv2D(256, kernel_size=(5, 5), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D(pool_size=(3, 3), strides=2))
    model.add(layers.Conv2D(384, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.Conv2D(384, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.Conv2D(256, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D(pool_size=(3, 3), strides=2))

    # Flatten the feature maps
    model.add(layers.Flatten())

    # Add Fully Connected layers
    model.add(layers.Dense(4096, activation='relu'))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(4096, activation='relu'))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(num_classes, activation='softmax'))

3. VGG

VGG (Visual Geometry Group) is another influential CNN architecture proposed by Karen Simonyan and Andrew Zisserman in 2014. It gained attention for its simplicity and achieved high accuracy in the ILSVRC 2014 competition.

Key Features of VGG:

Consists of 16-19 layers with a uniform architecture, using small 3x3 filters throughout the network.
Utilizes deep but narrower architectures compared to AlexNet.
Employs the ReLU activation function and batch normalization for faster convergence and reduced overfitting.
Uses max pooling for down-sampling.

Here's an example of building VGG using TensorFlow's Keras library:


    import tensorflow as tf
    from tensorflow.keras import layers, models

    # Create a Sequential model
    model = models.Sequential()

    # Add Convolutional layers
    model.add(layers.Conv2D(64, kernel_size=(3, 3), padding='same', activation='relu', input_shape=(image_height, image_width, num_channels)))
    model.add(layers.Conv2D(64, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=2))
    model.add(layers.Conv2D(128, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.Conv2D(128, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=2))
    model.add(layers.Conv2D(256, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.Conv2D(256, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.Conv2D(256, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=2))
    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=2))
    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=2))

    # Flatten the feature maps
    model.add(layers.Flatten())

    # Add Fully Connected layers
    model.add(layers.Dense(4096, activation='relu'))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(4096, activation='relu'))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(num_classes, activation='softmax'))

Common Mistakes in Understanding CNN Architectures

Using large kernel sizes without considering the computational cost.
Not adjusting the architecture for different input image sizes or tasks.
Overlooking the importance of proper weight initialization and regularization techniques.

Frequently Asked Questions

Q: What is the main difference between LeNet and AlexNet?
A: The main difference is the depth of the networks. LeNet is a relatively shallow architecture, while AlexNet is much deeper, allowing it to learn more complex features.
Q: Why is VGG called "Visual Geometry Group"?
A: VGG is named after the Visual Geometry Group at the University of Oxford, where the architecture was proposed by researchers Karen Simonyan and Andrew Zisserman.
Q: What are the advantages of using smaller filters in VGG?
A: Smaller filters (e.g., 3x3) help retain more spatial information, reduce the number of parameters, and increase the depth of the network, allowing it to learn more complex patterns.
Q: Can I use pre-trained weights for these architectures?
A: Yes, pre-trained weights for LeNet, AlexNet, and VGG are available in popular deep learning frameworks like TensorFlow and PyTorch. These pre-trained models can be used for transfer learning.
Q: Are these architectures suitable for tasks other than image classification?
A: While LeNet, AlexNet, and VGG were initially designed for image classification, their architectures and principles have been adapted for various computer vision tasks, including object detection and segmentation.

Summary

The architectures of Convolutional Neural Networks, such as LeNet, AlexNet, and VGG, have played a crucial role in advancing the field of computer vision. Each architecture brings unique features and design choices, making them suitable for different tasks and scenarios. Understanding these architectures can provide valuable insights for building and customizing deep learning models to achieve high accuracy and performance in various computer vision applications.