Activation functions help neural networks learn complex patterns. They take the input from one layer, apply a mathematical function, and pass the result to the next layer. This function enables the network to model non-linear relationships.
Why are they Essential?
Without activation functions, neural networks would be restricted to modeling only linear relationships between inputs and outputs.
At its core, a neural network computes the output of each neuron using the formula:
y = wx + b
Where:
w represents the weights,
x is the input,
b is the bias.
Without activation functions, no matter how many neurons or layers your network has, the entire model would simply perform a series of linear transformations. Even if you have 1,000 neurons or layers, the network would still only combine these calculations in a linear way. All layers would perform linear transformations of the input, and no non-linearities would be introduced.
Enabling Deep Architectures
✅Activation Functions are foundational in models ranging from basic multilayer perceptron to advanced architectures like GPT, enabling these networks to capture complex, non-linear dependencies in data.
A Visual Illustration
I’m presenting an illustration of the decision boundary on spiral data with and without activation functions.
Without Activation Functions:
The network computes only linear transformations, resulting in a straight-line decision boundary. This boundary is unable to capture the intricate structure of the spiral data, leading to poor performance.With Activation Function:
By applying the ReLU (Rectified Linear Unit) activation function, the network introduces non-linearity. This enables it to create a curved decision boundary that more accurately separates the spiral data into distinct classes. Showcasing the power of activation functions in learning complex patterns.
Liked this article? Make sure to 💜 click the like button.
Feedback or addition? Make sure to 💬 comment.
Know someone that would find this helpful? Make sure to 🔁 share this post.
Get in touch
You can find me on LinkedIn | YouTube | GitHub | X
Book an Appointment: Topmate
If you wish to make a request on particular topic you would like to read, you can send me an email to analyticalrohit.connect@gmail.com
Share this post