Utilizing Activation Capabilities in Neural Networks

Final Up to date on July 6, 2022

Activation capabilities play an integral position in neural networks by introducing non-linearity. This nonlinearity permits neural networks to develop advanced representations and capabilities based mostly on the inputs that may not be potential with a easy linear regression mannequin.

There have been many various non-linear activation capabilities proposed all through the historical past of neural networks. On this submit, we are going to discover three well-liked ones: sigmoid, tanh, and ReLU.

After studying this text, you’ll be taught:

  • Why nonlinearity is necessary in a neural community
  • How completely different activation capabilities can contribute to the vanishing gradient drawback
  • Sigmoid, tanh, and ReLU activation capabilities
  • The best way to use completely different activation capabilities in your TensorFlow mannequin

Let’s get began.

Utilizing Activation Capabilities in TensorFlow
Photograph by Victor Freitas. Some rights reserved.


This text is break up into 5 sections; they’re:

  • Why do we’d like nonlinear activation capabilities
  • Sigmoid operate and vanishing gradient
  • Hyperbolic tangent operate
  • Rectified Linear Unit (ReLU)
  • Utilizing the activation capabilities in apply

Why Do We Want Nonlinear Activation Capabilities

You could be questioning, why all this hype about non-linear activation capabilities? Or why can’t we simply use an id operate after the weighted linear mixture of activations from the earlier layer. Utilizing a number of linear layers is principally the identical as utilizing a single linear layer. This may be seen by a easy instance. Let’s say now we have a one hidden layer neural community, every with two hidden neurons.

Single hidden layer neural community with linear layers

We will then rewrite the output layer as a linear mixture of the unique enter variable if we used a linear hidden layer. If we had extra neurons and weights, the equation can be loads longer with extra nesting and extra multiplications between successive layer weights however the concept stays the identical: we will signify all the community as a single linear layer. To make the community to signify extra advanced capabilities, we would wish nonlinear activation capabilities. Let’s begin with a preferred instance, the sigmoid operate.

Sigmoid Operate and Vanishing Gradient

The sigmoid activation operate is a well-liked alternative for the non-linear activation operate for neural networks. One purpose for its reputation is that it has output values between 0 and 1 which mimic likelihood values and is therefore used to transform the true valued output of a linear layer to a likelihood, which can be utilized as a likelihood output. This has additionally allowed it to be an necessary a part of logistic regression strategies which can be utilized instantly for binary classification.

The sigmoid operate is often represented by $sigma$ and has the shape $sigma = frac{1}{1 + e^{-1}}$. In TensorFlow, we will name the sigmoid operate from the Keras library as follows:

This offers us the output:

We will additionally plot the sigmoid operate as a operate of $x$,

Sigmoid activation operate

When trying on the activation operate for the neurons in a neural community, we must also be involved in its spinoff resulting from backpropagation and the chain rule which might have an effect on how the neural community learns from information.

Sigmoid activation operate (blue) and gradient (orange)

Right here, we will observe that the gradient of the sigmoid operate is all the time between 0 and 0.25. And because the $x$ tends to optimistic or destructive infinity, the gradient tends to zero. This might contribute to the vanishing gradient drawback, which when the enter are at some massive magnitude of $x$ (e.g., as a result of output from earlier layers), the gradient is just too small to provoke the correction.

Vanishing gradient is an issue as a result of we use the chain rule in backpropagation in deep neural networks. Recall that in neural networks, the gradient (of the loss operate) at every layer is the gradient at its subsequent layer multiplied with the gradient of its activation operate. As there are various layers within the community, if the gradient of the activation capabilities are lower than 1, the gradient at some layer distant from output will probably be near zero. And any layer with a gradient near zero will cease the gradient propagate additional again to the sooner layers.

Because the sigmoid operate is all the time lower than 1, a community with extra layers would exacerbate the vanishing gradient drawback. Moreover, there’s a saturation area the place the gradient of the sigmoid tends to 0, which is the place the magnitude of $x$ is massive. So, if the output of the weighted sum of activations from earlier layers is massive then we might have a really small gradient propagating by this neuron because the spinoff of the activation $a$ with respect to the enter to the activation operate can be small (in saturation area).

Granted, there may be additionally the spinoff of the linear time period with respect to the earlier layer’s activations which could be higher than 1 for the layer because the weight could be massive and it’s a sum of derivatives from the completely different neurons. Nonetheless, it’d nonetheless elevate concern firstly of coaching as weights are often initialized to be small.

Hyperbolic Tangent Operate

One other activation operate we will contemplate is the tanh activation operate, in any other case referred to as the hyperbolic tangent operate. It has a bigger vary of output values in comparison with the sigmoid operate and has a bigger most gradient as properly. The tanh operate is hyperbolic analogue to the traditional tangent operate for circles that most individuals are aware of.

Plotting out the tanh operate,

Tanh activation operate

Let’s have a look at the gradient as properly,

Tanh activation operate (blue) and gradient (orange)

Discover that the gradient now has a most worth of 1, in comparison with the sigmoid operate the place the most important gradient worth is at 0. This makes a community with tanh activation much less inclined to the vanishing gradient drawback. Nonetheless, the tanh operate additionally has a saturation area, the place the worth of the gradient tends in direction of because the magnitude of the enter $x$ will get bigger.

In TensorFlow, we will implement the tanh activation on a tensor utilizing the tanh operate in Keras’ activations module

which provides the output

Rectified Linear Unit (ReLU)

The final activation operate we’ll have a look at intimately is the Rectified Linear Unit, additionally popularly referred to as ReLU. It has grow to be well-liked just lately resulting from its comparatively easy computation which helps to hurry up neural networks and appears to get empirically good efficiency, which makes it a very good beginning alternative for the activation operate.

The ReLU operate is a straightforward $max(0, x)$ operate, which will also be regarded as a piecewise operate with all inputs lower than 0 mapping to 0 and all inputs higher than or equal to 0 mapping again to themselves (i.e., id operate). Graphically,

ReLU activation operate

Subsequent up, we will additionally have a look at the gradient of the ReLU operate:

ReLU activation operate (blue line) and gradient (orange)

Discover that the gradient of ReLU is 1 each time the enter is optimistic, which is useful in addressing the vanishing gradient drawback. Nonetheless, each time the enter is destructive, the gradient is 0 which might trigger one other drawback, the lifeless neuron/dying ReLU drawback, which is a matter if a neuron is persistently inactivated. On this case, the neuron is rarely in a position to be taught and its weights are by no means up to date as a result of chain rule because it has a 0 gradient as one in every of its phrases. If this occurs for all information in your dataset then it may be very troublesome for this neuron to be taught out of your dataset until the activations within the earlier layer change such that the neuron is now not “lifeless”.

To make use of the ReLU activation in TensorFlow,

which provides us the output:

Over the three activation capabilities we reviewed above, we see that they’re all monotonically rising capabilities. That is required or in any other case we can’t apply the gradient descent algorithm.

Now that we’ve explored some widespread activation capabilities and methods to use them in TensorFlow, let’s check out how we will use these in apply in an precise mannequin.

Utilizing Activation Capabilities in Apply

Earlier than we discover using activation capabilities in apply, let’s have a look at one other widespread manner that we will use activation capabilities when combining them with one other Keras layer. Let’s say we wish to add a ReLU activation on prime of a Dense layer. A technique we will do that following the above strategies proven is to do

Nonetheless, for a lot of Keras layers, we will additionally use a extra compact illustration so as to add the activation on prime of the layer:

Utilizing this extra compact illustration, let’s construct our LeNet5 mannequin utilizing Keras:

And working this code provides us the output

And that’s how we will use completely different activation capabilities in our TensorFlow fashions!

Additional Studying

Different examples of activation capabilities:


On this submit, you will have seen why activation capabilities are necessary to permit for the advanced neural networks that we see widespread in deep studying at this time. You may have additionally seen some well-liked activation capabilities, their derivatives, and methods to combine them into your TensorFlow fashions.

Particularly, you discovered:

  • Why non-linearity is necessary in a neural community
  • How completely different activation capabilities can contribute to the vanishing gradient drawback
  • Sigmoid, tanh, and ReLU activation capabilities
  • The best way to use completely different activation capabilities in your TensorFlow mannequin

Develop Deep Studying Tasks with Python!

Deep Learning with Python

 What If You May Develop A Community in Minutes

…with just some strains of Python

Uncover how in my new E book:

Deep Studying With Python

It covers end-to-end initiatives on subjects like:

Multilayer PerceptronsConvolutional Nets and Recurrent Neural Nets, and extra…

Lastly Convey Deep Studying To

Your Personal Tasks

Skip the Teachers. Simply Outcomes.

See What’s Inside

Leave a Reply