Models

Model Selection

The selection of model architecture for the agent can be made in the agent configuration file.

Currently, there exists two fully connected neural network models one for the actor and one for the critic available.

• --actor_model: Actor model (default='actor_fcnn-v0')
• --critic_model: Critic model (default='critic_fcnn-v0')

ActorFCNN / CriticFCNN Architecture

The ActorFCNN/CriticFCNN class, inheriting from a custom Model base class, implements a fully connected neural network (FCNN) designed specifically for use in actor-critic reinforcement learning algorithms.

Key Features

• Configuration Loading: The model architecture supports loading configurations from a specified .cfg file, enabling easy adjustments to model parameters without altering the codebase.

• Logging: Utilizes the logging module to provide debug and error messages, particularly useful for tracking the model's configuration saving process.

• Actor Layer Architecture:

  • Three dense layers with the first two featuring 256 units and ReLU activation. These layers are responsible for processing the input state.
  • The third layer outputs actions scaled to the environment's action space, using a tanh activation function to ensure output values fall within a normalized range.

• Critic Layer Architecture:

  • A neural network with three dense layers, designed to process concatenated state and action inputs.
  • The first two layers have 256 units each with ReLU activation, responsible for extracting features from the input data.
  • The final layer outputs a single value, representing the estimated Q-value of the input state-action pair.

• Action Scaling and Biasing: Outputs from the final actor layer are scaled and biased to match the environment's actual action range, utilizing provided min_action and max_action parameters.

• Configuration Saving: Includes functionality to save the model configuration to the specified logdir, ensuring that the model setup can be easily replicated or reviewed.

The ActorFCNN/CriticFCNN model is designed for applications in reinforcement learning where an actor is required to decide on actions given the current state of the environment and the critic is designed to estimate the q values of the given state action. Its modular design and external configuration support make it flexible for various tasks and easy to adjust according to different requirements.

Model Base Class Documentation

The Model class extends tf.keras.Model, making it a specialized abstract base class (ABC) for creating machine learning models with TensorFlow. This class is designed to serve as a foundation for defining the architecture of various deep learning models. It provides a structured way to implement custom models by specifying the forward pass behavior and configuration management.

Constructor

__init__(self, **kwargs)

Initializes a new instance of the Model class. This constructor is designed to define all key variables required for all models, leveraging the initialization process of tf.keras.Model through super().__init__(**kwargs).

Parameters:

• **kwargs: Arbitrary keyword arguments that are passed to the parent tf.keras.Model constructor. This allows for the flexible configuration of the model, including naming the model, setting up layers, and other TensorFlow-specific settings.

Methods

call(self, inputs, training=False)

An abstract method that must be implemented by subclasses. This method defines the forward pass of the model. It determines how the model processes input data and returns the output. The training parameter indicates whether the model should behave in training mode or inference mode.

Parameters:

• inputs: The input tensor(s) to the model.
• training: Boolean flag indicating whether the model is in training mode. Default is False.

Returns:

• The output tensor(s) of the model after the forward pass.

save_cfg(self)

An abstract method that must be implemented by subclasses. This method should save the configuration of the model, such as its architecture, hyperparameters, and any other relevant settings. This is crucial for reproducing the model's behavior, sharing models, and maintaining model versioning.

Implementing a Custom Model

To create a specific machine learning model (e.g., a convolutional neural network for image classification, a recurrent neural network for sequence processing), one must subclass Model and provide concrete implementations for all the abstract methods. This involves defining the layers and operations that constitute the model's architecture in the call method and detailing how the model's configuration can be saved and potentially reloaded.