The activation function is an integral part of a neural network. A neural network is a simple linear regression model without an activation function. This means the activation function gives non-linearity to the neural network gradient parameter. In this article, we will discuss the SoftMax activation function, which is popularly used for multiclass classification problems. Let’s first understand the neural network architecture for a multiclass classification problem and why other activation functions can not be used in this case.
The SoftMax activation function is commonly used in machine learning, particularly in neural networks for classification tasks. An activation function converts a vector of raw prediction scores (logits) into probabilities.
Suppose we have the following dataset: For every observation, we have five features from FeatureX1 to FeatureX5, and the target variable has three classes.
Now, let’s create a simple neural network to solve this problem. Here, we have an Input layer with five neurons, as we have five features in the dataset. Next, we have one hidden layer with four neurons. Each of these neurons uses inputs, weights, and biases to calculate a value, which is represented as Zij here.
For example, the first neuron of the first layer is represented as Z11 Similarly the second neuron of the first layer is represented as Z12, and so on.
We apply the activation function, let’s say a tanh activation function, to these values and send the values or result to the output layer.
The number of neurons in the output layer depends on the number of classes in the dataset. Since we have three classes in the dataset, we will have three neurons in the output layer. Each of these neurons will give the probability of individual classes. This means the first neuron will give you the probability that the data point belongs to class 1. Similarly, the second neuron will give you the probability that the data point belongs to class 2.
Also Read: Unlocking The Power of Activation Functions in Neural Networks
Here’s how the softmax function in machine learning works in the last layer of a neural network :
Input: The softmax activation function takes a vector of real numbers (z) as input. These values typically represent the outputs from the final hidden layer of the neural network, often accessed via an API.
Exponentiation: Each element in the input vector z is exponentiated using the mathematical constant e (approximately 2.718). This step ensures all the values become positive. The derivative of this step is crucial for backpropagation.
Normalization: After exponentiation, all the elements are summed up. This is a key step for ensuring that the probabilities add up to 1.
Probability Calculation: Each exponentiated value from step 2 is then divided by the sum obtained in step 3. This process normalizes the values, forcing them between 0 and 1. The cross-entropy loss function often uses these probabilities to measure a classifier’s performance.
Output: The result is a new vector the same size as the input vector z. However, each element in the output vector now represents a probability between 0 and 1. The argmax function is typically used to select the index of the highest probability, determining the predicted class generalization.
Suppose we calculate the Z value using weights and biases of this layer and apply the sigmoid activation function over these values. We know that the sigmoid activation function gives the value between 0 and 1 suppose these are the values we get as output.
There are two problems in this case-
First, if we apply a threshold of 0.5, this network says the input data point belongs to two classes. Secondly, these probability values are independent of each other. That means the probability that the data point belongs to class 1 does not consider the probability of the other two classes.
The sigmoid activation function is not preferred in multi-class classification problems.
In the above example, we will use the Softmax activation function in the output layer instead of sigmoid. The Softmax activation function calculates relative probabilities. That means it uses the values of Z21, Z22, and Z23 to determine the final probability value.
Let’s see how the softmax activation function works. Like the sigmoid activation function, the SoftMax function in machine learning returns the probability of each class. Here is the equation for the SoftMax activation function.
Here, Z represents the values from the neurons of the output layer. The exponential acts as the non-linear function. Later, these values are divided by the sum of exponential values to normalize them and then convert them into probabilities.
Note that, when the number of classes is two, it becomes the same as the sigmoid activation function. In other words, sigmoid is simply a variant of the Softmax function. To learn more about this concept, refer to this link.
Let’s understand with a simple example how the softmax function works. We have the following neural network.
Suppose the values of Z21, Z22, and Z23 are 2.33, -1.46, and 0.56, respectively. Now, the SoftMax activation function is applied to each of these neurons, and the following values are generated.
These are the probability values that a data point belonging to the respective classes. Note that, in this case, the sum of the probabilities is equal to 1.
In this case, the input belongs to class 1. So, if the probability of any of these classes is changed, the probability value of the first class would also change.
Softmax Function is typically used in the last layer of a neural network to predict the class of an input image. It is also used in other applications, such as natural language processing and machine translation.
ReLU is typically used in the hidden layers of a neural network to add non-linearity. It is efficient and can help neural networks learn more complex relationships between the input and output data.
Here is how Softmax function in machine learning is used in CNN :
In machine learning, functions like softmax output are implemented in frameworks such as numpy and Python to facilitate the process. The softmax function, through exponentiation, transforms the logits into a probability distribution. This method is crucial in determining the loss function during model training and optimization. CNN’s ability to make precise predictions hinges on these fundamental principles.
This article is all about the SoftMax activation function. In it, we saw why we should not use activation functions like sigmoid or threshold in multiclass classification problems and how the SoftMax function works through an example. In this article, these algorithms will show different SoftMax output values and different output vectors.
The softmax activation function is a mathematical function that converts a vector of real numbers into a probability distribution. It exponentiates each element, making them positive, and then normalizes them by dividing by the sum of all exponentiated values. This ensures that the output probabilities add up to one, making it suitable for multiclass classification tasks.
The sigmoid function is used for binary classification, mapping any real value to a range between 0 and 1. It’s suitable for independent predictions. The softmax function in machine learning, on the other hand, converts a vector of real numbers into a probability distribution for multiclass classification tasks, ensuring that the sum of the probabilities is equal to one
If you want to kick-start your Data Science Journey and want every topic under one roof, your search stops here. Check out Analytics Vidhya’s Certified AI & ML BlackBelt Plus Program
Lorem ipsum dolor sit amet, consectetur adipiscing elit,