Neural Network Optimization Algorithms

Optimizing the weights and baises of neural networks using genetic algorithm and particle swarm optimization methods. Includes custom PyTorch optimizers.
AI / Machine Learning
Author

Brandon Toews

Published

May 23, 2024

Video Walkthrough
Project Description

This project aims to compare the performance of various neural network models and optimization algorithms on different datasets. The models include standard neural networks, quantized neural networks, and models implemented in PyTorch. The optimization algorithms include Genetic Algorithm (GA) and Particle Swarm Optimization (PSO).

Source Code

View source code on Github repository.

1 Models and Optimizers

1.1 Custom Models

  • Neural Network: A basic feedforward neural network implemented from scratch.
  • Quantized Neural Network: A neural network with quantized weights for improved efficiency.

1.2 PyTorch Models

  • XOR_Model: A model for the XOR problem using Adam optimizer.
  • XOR_GA_Model: A model for the XOR problem using Genetic Algorithm.
  • XOR_PSO_Model: A model for the XOR problem using Particle Swarm Optimization.
  • Mnist_Model: A model for the MNIST dataset using Adam optimizer.
  • Mnist_GA_Model: A model for the MNIST dataset using Genetic Algorithm.
  • Mnist_PSO_Model: A model for the MNIST dataset using Particle Swarm Optimization.

1.3 Custom Optimizers

1.3.1 Genetic Algorithm

  • Functions:
    • __init__: Initializes the optimizer with a given model, population size, mutation rate, and weight range.
    • _get_model_parameters: Retrieves the model parameters as a list of numpy arrays.
    • _set_model_parameters: Sets the model parameters from a given list of numpy arrays.
    • generate_population: Generates an initial population of solutions with random weights.
    • fitness_function: Calculates the fitness of a model based on the inverse of the loss function.
    • layer_crossover: Performs uniform crossover between two parent solutions at the layer level.
    • neuron_crossover: Performs uniform crossover between two parent solutions at the neuron level.
    • mutate: Mutates a given solution by randomly altering its weights.
    • step: Executes one iteration of the genetic algorithm, involving evaluation of the population, selection of the best individuals, and creation of a new population through crossover and mutation.
  • Usage in Models:
    • Mnist_GA_Model: Trains a PyTorch model for the MNIST dataset.
    • XOR_GA_Model: Trains a PyTorch model for the XOR problem.
    • Quantized_NN: Trains a custom quantized neural network.

1.3.2 Particle Swarm Optimization

  • Functions:
    • __init__: Initializes the optimizer with a given model, weight range, number of particles, and clipping option.
    • _get_model_parameters: Retrieves the model parameters as a list of numpy arrays.
    • _set_model_parameters: Sets the model parameters from a given list of numpy arrays.
    • generate_particles: Generates a swarm of particles with random positions and velocities.
    • objective_function: Calculates the objective value (loss) of a model.
    • clip_move: Moves a particle with clipping.
    • move: Moves a particle without clipping.
    • step: Executes one iteration of the particle swarm optimization algorithm, involving updating the velocity and position of each particle based on cognitive and social components.
  • Usage in Models:
    • Mnist_PSO_Model: Trains a PyTorch model for the MNIST dataset.
    • XOR_PSO_Model: Trains a PyTorch model for the XOR problem.
    • Quantized_NN: Trains a custom quantized neural network.

2 Results 📊

Results are visualized using Matplotlib to compare the training loss and accuracy of each model over epochs. The performance of different optimization algorithms is compared in terms of training time, final loss, and accuracy.

2.1 XOR Problem

XOR PyTorch Results XOR Normal vs Quantized GA Results

2.2 MNIST Dataset

  • Adam: Mnist PyTorch with Adam Results
  • Genetic Algorithm: Mnist GA Results
  • Particle Swarm Optimization: Mnist PSO Results