{"id":246993,"date":"2026-04-30T09:51:08","date_gmt":"2026-04-30T14:51:08","guid":{"rendered":"https:\/\/www.johndcook.com\/blog\/?p=246993"},"modified":"2026-04-30T13:39:40","modified_gmt":"2026-04-30T18:39:40","slug":"derivative-of-relu","status":"publish","type":"post","link":"https:\/\/www.johndcook.com\/blog\/2026\/04\/30\/derivative-of-relu\/","title":{"rendered":"Three ways to differentiate ReLU"},"content":{"rendered":"

When a function is not differentiable in the classical sense there are multiple ways to compute a generalized derivative. This post will look at three generalizations of the classical derivative, each applied to the ReLU (rectified linear unit) function. The ReLU function is a commonly used activation function<\/a> for neural networks. It’s also called the ramp function for obvious reasons.<\/p>\n

<\/p>\n

The function is simply\u00a0r<\/em>(x<\/em>) = max(0,\u00a0x<\/em>).<\/p>\n

Pointwise derivative<\/h2>\n

The pointwise<\/strong> derivative would be 0 for\u00a0x<\/em> < 0, 1 for x<\/em> > 0, and undefined at\u00a0x<\/em> = 0. So except at 0, the pointwise derivative of the ramp function is the Heaviside<\/a> function.<\/p>\n

\"H(x)
\nIn a real analysis course, you’d simply say\u00a0r<\/em>\u2032(x<\/em>) =H<\/em>(x<\/em>) because functions are only defined up to equivalent modulo sets of measure zero, i.e. the definition at\u00a0x<\/em> = 0 doesn’t matter.<\/p>\n

Distributional derivative<\/h2>\n

In\u00a0distribution theory<\/strong> you’d identify the function\u00a0r<\/em>(x<\/em>) with the distribution whose action on a test function \u03c6 is<\/p>\n

\"\\langle<\/p>\n

Then the derivative of r<\/em> would be the distribution r<\/em>\u2032 satisfying<\/p>\n

\"\\langle<\/p>\n

for all smooth functions \u03c6 with compact support. You can prove using integration by parts that the above equals the integral of \u03c6 from 0 to \u221e, which is the same as the action of H<\/em>(x<\/em>) on \u03c6.<\/p>\n

In this case the distributional derivative of\u00a0r<\/em> is the same as the pointwise derivative of\u00a0r<\/em> interpreted as a distribution. This does not happen in general. For example, the pointwise derivative of\u00a0H<\/em> is zero but the distributional derivative of H<\/em> is \u03b4, the Dirac delta distribution.<\/p>\n

For more on distributional derivatives, see How to differentiate a non-differentiable function<\/a>.<\/p>\n

Subgradient<\/h2>\n

The subgradient of a function f<\/em> at a point x<\/em>, written \u2202f<\/em>(x<\/em>), is the set of slopes of tangent lines to the graph of\u00a0f<\/em> at\u00a0x<\/em>. If\u00a0f<\/em> is differentiable at\u00a0x<\/em>, then there is only one slope, namely\u00a0f<\/em>\u2032(x<\/em>), and we typically say the subgradient of f<\/em> at\u00a0x<\/em> is simply f<\/em>\u2032(x<\/em>) when strictly speaking we should say it is the one-element set {f<\/em>\u2032(x<\/em>)}.<\/p>\n

A line tangent to the graph of the ReLU function at a negative value of x<\/em> has slope 0, and a tangent line at a positive\u00a0x<\/em> has slope 1. But because there’s a sharp corner at\u00a0x<\/em> = 0, a tangent at this point could have any slope between 0 and 1.<\/p>\n

\"\\partial<\/p>\n

My dissertation was full of subgradients of convex functions. This made me uneasy because subgradients are not real-valued functions; they’re set-valued functions. Most of the time you can blithely ignore this distinction, but there’s always a nagging suspicion that it’s going to bite you unexpectedly.<\/p>\n","protected":false},"excerpt":{"rendered":"

When a function is not differentiable in the classical sense there are multiple ways to compute a generalized derivative. This post will look at three generalizations of the classical derivative, each applied to the ReLU (rectified linear unit) function. The ReLU function is a commonly used activation function for neural networks. It’s also called the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[9],"tags":[],"class_list":["post-246993","post","type-post","status-publish","format-standard","hentry","category-math"],"acf":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.johndcook.com\/blog\/wp-json\/wp\/v2\/posts\/246993","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.johndcook.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.johndcook.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.johndcook.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.johndcook.com\/blog\/wp-json\/wp\/v2\/comments?post=246993"}],"version-history":[{"count":0,"href":"https:\/\/www.johndcook.com\/blog\/wp-json\/wp\/v2\/posts\/246993\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.johndcook.com\/blog\/wp-json\/wp\/v2\/media?parent=246993"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.johndcook.com\/blog\/wp-json\/wp\/v2\/categories?post=246993"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.johndcook.com\/blog\/wp-json\/wp\/v2\/tags?post=246993"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}