The mistake I kept making
The derivative of ReLU is 1 when the input is positive, 0 when it is negative, and it has no single value at exactly zero. That one line hides a lesson I learned the hard way.
While practising piecewise functions, I kept making the same error. A question would ask for the value of the function as I approached a point, and I would plug the number into the derivative instead of the original function. I was answering a slope question when I was asked about height. My mentor kept catching it, and I kept repeating it.
In my previous post I wrote that derivatives are the language neural networks speak. This post is about the most popular piece of that language, ReLU, and how fixing my height vs slope mistake is exactly what made ReLU make sense to me.
Height and slope answer two different questions
Every function can be asked two separate questions at any point:
- Where am I? That is the height, and the original function f(x) answers it.
- Which way am I moving, and how fast? That is the slope, and the derivative f′(x) answers it.
Take f(x) = x² at x = 3. The height is 9. The slope is 2 × 3 = 6. Both numbers describe the same point, but they mean completely different things. Mixing them up is like telling someone your car is at kilometre 60 when they asked how fast you are driving.
Keep that picture in mind, because ReLU is a function where the height and the slope behave very differently.
ReLU is just a piecewise function
ReLU stands for Rectified Linear Unit. The name sounds heavy, but the rule is simple: if the input is positive, pass it through unchanged. If it is negative, output zero.
The kink at zero
Zero is where my height vs slope lesson really pays off, because it asks two separate questions with two separate answers.
First question: is ReLU continuous at zero? This is a height question, so we use the original function. Coming from the left, the height approaches 0. Coming from the right, the height also approaches 0. And ReLU(0) is 0. All three match, so there is no jump. ReLU is continuous.
Second question: does ReLU have a derivative at zero? This is a slope question. From the left the slope is 0. From the right the slope is 1. The two sides disagree, so there is no single slope at that point. The graph has a sharp corner, a kink, and that makes ReLU not differentiable at exactly zero.
This is the exact trap I used to fall into. If you check continuity with the slopes, you wrongly conclude there is a break. If you check differentiability with the heights, you wrongly conclude everything is smooth. Heights for continuity, slopes for differentiability.
So how does a neural network train if ReLU has no derivative at zero? In practice, libraries simply pick a value for that one point, commonly 0. An input landing on exactly 0.0 is rare with real numbers, and choosing either slope there works fine.
What ReLU's slope does during learning
During the forward pass, the network uses the height of ReLU to make a prediction. During backpropagation, it uses only the slope. That split is the whole story.
Picture a single neuron whose input to ReLU is z. The learning signal arriving from later layers gets multiplied by ReLU′(z) before it moves backwards.
- If z = 3, the height is 3 and the slope is 1. The learning signal passes through untouched, so the weights behind this neuron get updated.
- If z = −2, the height is 0 and the slope is 0. The learning signal is multiplied by 0 and stops. The weights behind this neuron learn nothing from this example.
This simple on and off switch is a big reason ReLU became so popular. Older activations like the sigmoid have a slope of at most 0.25, so the signal shrinks a little at every layer and fades away in deep networks. A slope of exactly 1 lets the signal travel through many layers without fading.
The switch has a dark side too. If a neuron's input stays negative for every example, its slope is always 0 and it never learns again. This is called the dying ReLU problem, and it is why variants like Leaky ReLU give the negative side a small slope instead of zero.
My 10 second rule
Before I plug any number into anything, I now ask myself one question: is this asking where the function is, or how it is moving?
Where it is means height, so I use the original function. How it is moving means slope, so I use the derivative. Ten seconds of thinking has saved me from the same mistake again and again.
The funny part is that a neural network lives by this exact split. It predicts with heights and learns with slopes. Fixing a beginner mistake in my notebook ended up teaching me how the most common activation function in deep learning actually works.
If you are learning math for AI, do not hide your mistakes. Study them. Some of them turn out to be the best teachers you will have. Next in this series, I will look at the chain rule and how it carries these slopes backwards through an entire network.