Skip to article
← All posts

Deep learning and differential equations

Training (very) deep ResNets

What differential equations tell us about training residual networks.

The understanding presented here grew out of Raphaël Barboni’s work on training residual networks. In particular, the beautiful diagonal analysis—using a commuting matrix family to reveal a complex-variable loss—and the resulting two-dimensional plots are due to him.

Residual networks let a layer learn a correction: keep the current activation and add an update. These skip connections made much deeper networks practical [1], [2]; they are also central to transformer architectures.

The identity path helps information and gradients travel through the network. Within an identity-skip block, the input and output must have the same dimension. This does not fix the width of the whole architecture: projection shortcuts, embeddings, and changes between stages can change dimensions. Here we keep a fixed state space and study the limit of many small residual steps.

This limit connects network training to differential equations and optimal control. Stable propagation alone, however, does not guarantee easy optimization. An exactly solvable example below has no nonglobal local minima, but its saddles become flatter as depth increases.

Residual networks and the ODE limit

Let \(x_\ell\in\mathbb R^d\) be the activation, \(\theta_\ell\in\mathbb R^p\) the parameters of block \(\ell\), and \(\phi:\mathbb R^d\times\mathbb R^p\to\mathbb R^d\) a residual function. We assume \(\phi\) is continuously differentiable for the gradient calculations below. With depth \(L\), write

\[h=\frac1L,\qquad x_{\ell+1}=x_\ell+h\phi(x_\ell,\theta_\ell), \qquad \ell=0,\ldots,L-1.\]
One residual blockThe activation travels directly to a sum along an identity path. A second branch applies phi and scales its result by one over L before joining the sum.xℓφ(xℓ, θℓ) / L+xℓ₊₁identity shortcut
The two branches must have the same output dimension for their sum to make sense.

This is the explicit Euler scheme on a time interval of length one. If the weights sample a sufficiently regular profile \(\theta_\ell=\theta(\ell/L)\), the depth variable becomes a continuous variable \(t\):

\[\partial_t x(t)=\phi(x(t),\theta(t)),\qquad x(0)=a, \qquad 0\le t\le1.\]

For example, a Lipschitz parameter profile, a smooth \(\phi\), and uniform bounds on the relevant trajectories and derivatives give the usual Euler error \(O(L^{-1})\). The dynamical-systems and control viewpoints were developed in [3]–[6].

The factor \(1/L\) is a deliberate ODE scaling, not part of every practical ResNet. Arbitrarily oscillating or independently initialized layer weights do not automatically converge to a single smooth profile. Other scalings and initializations lead to different limits; convergence of a trained network is a further question [7], [8].

Proposition 1 (depth-independent propagation bounds). Suppose \(\|D_x\phi(x_\ell,\theta_\ell)\|_{\mathrm{op}}\le K\) for every layer, with \(K\) independent of \(L\). The Jacobian of the network with respect to its input satisfies

\[\|D_a x_L\|_{\mathrm{op}} \le(1+K/L)^L\le e^K.\]

If also \(L>K\), its smallest singular value obeys

\[\sigma_{\min}(D_a x_L)\ge(1-K/L)^L.\]

The lower bound tends to \(e^{-K}>0\). Thus, under a uniform derivative bound, merely increasing the depth neither explodes nor extinguishes gradients propagated through the state. The constants can still be poor when \(K\) is large; this does not control parameter gradients.

Proof: multiply near-identity Jacobians

Each block has Jacobian \(I+hD_x\phi\), of norm at most \(1+hK\). Apply the chain rule and submultiplicativity. For the lower bound, \(\|(I+hB)v\|\ge(1-hK)\|v\|\) whenever \(\|B\|_{\mathrm{op}}\le K\); iterate this inequality over the blocks. The same singular-value bounds apply to the transposed product used in backpropagation. These are propagation estimates, not a theorem that every parameter is learnable or every loss easy to minimize. \(\square\)

Gradient descent and depth-dependent scaling

Take training pairs \((a_i,b_i)_{i=1}^N\), with a differentiable terminal loss \(\ell(x,b)\). A fixed output head can be absorbed into \(\ell\). Let \(x^i_0=a_i\), choose \(\lambda\ge0\), and consider

\[\mathcal J_L(\theta)=\frac1N\sum_{i=1}^N\ell(x^i_L,b_i) +\frac{\lambda h}{2}\sum_{\ell=0}^{L-1}\|\theta_\ell\|^2.\]

The index \(\ell\) counts layers; the variable \(s\) below measures training time. At each training time, a forward pass computes all the states. A backward pass computes the adjoints

\[\begin{aligned} p^i_L&=\nabla_x\ell(x^i_L,b_i),\\ p^i_\ell&=\bigl(I+hD_x\phi(x^i_\ell,\theta_\ell)\bigr)^\top p^i_{\ell+1}. \end{aligned}\]

Ordinary backpropagation now gives the exact Euclidean gradient:

\[\nabla_{\theta_\ell}\mathcal J_L =h\left[\frac1N\sum_i D_\theta\phi(x^i_\ell,\theta_\ell)^\top p^i_{\ell+1} +\lambda\theta_\ell\right].\]

The factor \(h\) matters. Ordinary gradient flow is \(d\theta_\ell/ds=-\nabla_{\theta_\ell}\mathcal J_L\). To compare different depths on the same nontrivial training clock, use instead

\[\frac{d\theta_\ell}{ds}=-L\nabla_{\theta_\ell}\mathcal J_L.\]

This is gradient flow for the discrete depth metric \(\langle u,v\rangle_L=h\sum_\ell u_\ell\cdot v_\ell\), which approximates the \(L^2\) metric on parameter paths. In gradient descent, it corresponds to updates \(\theta_\ell\leftarrow\theta_\ell-\eta L\nabla_{\theta_\ell}\mathcal J_L\). It is a normalization of the model and time scale—not a recommendation to multiply an arbitrary implementation's learning rate by its depth.

Backpropagation in continuous depth

Fix a bounded measurable parameter path \(\theta\), with state trajectories existing on \([0,1]\). The continuum objective is

\[\mathcal J(\theta)=\frac1N\sum_i\ell(x^i(1),b_i) +\frac\lambda2\int_0^1\|\theta(t)\|^2\,dt.\]

Proposition 2 (the adjoint first variation). Assume \(\phi\) and the loss are continuously differentiable, with their derivatives bounded on a neighborhood of the state and parameter values under consideration. Solve the state equation forward and the following adjoint backward, with the differential equations understood almost everywhere:

\[-\partial_t p^i(t)=D_x\phi(x^i(t),\theta(t))^\top p^i(t), \qquad p^i(1)=\nabla_x\ell(x^i(1),b_i).\]

For every bounded perturbation \(u\), the directional derivative is \(D\mathcal J(\theta)[u]=\int_0^1G(t)\cdot u(t)\,dt\), where

\[G(t)=\frac1N\sum_i D_\theta\phi(x^i(t),\theta(t))^\top p^i(t) +\lambda\theta(t).\]
Proof: the adjoint equation and integration by parts

For a perturbation \(u\), the state variation \(\xi^i\) satisfies

\[\partial_t\xi^i=D_x\phi\,\xi^i+D_\theta\phi\,u, \qquad \xi^i(0)=0.\]

The adjoint cancels the term involving \(D_x\phi\):

\[\partial_t(p^i\cdot\xi^i)=p^i\cdot D_\theta\phi\,u.\]

Integrate from zero to one, use the terminal value of \(p^i\), average over the data, and add the derivative of the quadratic penalty. Standard dependence of ODE solutions on parameters justifies this linearization for bounded \(u\) and sufficiently small perturbation amplitude. The bounded coefficients and finite time interval make \(G\) bounded, hence square-integrable. \(\square\)

We call \(G\) the \(L^2\)-gradient: it represents the first variation in the depth inner product. This does not, under these local assumptions alone, assert that the objective is defined and differentiable on an open set of all \(L^2\) controls. The corresponding training equation evolves an entire bounded depth profile:

\[\partial_s\theta_s(t)=-G_s(t).\]

As long as a sufficiently regular bounded training trajectory exists, the chain rule gives the energy identity

\[\frac{d}{ds}\mathcal J(\theta_s) =-\int_0^1\|G_s(t)\|^2\,dt\le0.\]

For each \(s\), the states and adjoints must be recomputed across \(t\). Think of a forward–backward sweep inside each training step. Proving that discrete training converges to this flow requires additional uniform control, not just convergence of a single forward pass [10], [11]. Variational convergence of objectives and minimizers is a different question: the result in [7] uses explicit regularization across neighboring layers, not merely the quadratic weight penalty written here. Also, at a fixed numerical step size, solving the continuous adjoint is not automatically identical to differentiating a discretized solver [6].

Can training preserve the smoothness of the weights across depth? In work with Michael Sander and Pierre Ablin, we studied this question for linear residual networks under suitable initialization and small-loss assumptions [10]. Marion, Wu, Sander, and Biau developed a nonlinear analysis with smooth activations: finite-time control uses boundedness/clipping hypotheses, and long-time convergence uses a Polyak–Łojasiewicz condition [11]. Neither statement follows from skip connections alone; the simple quadratic weight penalty above does not itself enforce smoothness across depth.

Linear residual networks and time-ordered exponentials

Now set \(\phi(x,A)=Ax\), with \(A\in\mathbb R^{d\times d}\). The finite network and the limiting flow are

\[x_L=(I+hA_{L-1})\cdots(I+hA_0)a, \qquad \partial_t x=A(t)x.\]

Every initial condition shares the same fundamental matrix:

\[x(t)=\Phi(t)a,\qquad \partial_t\Phi(t)=A(t)\Phi(t),\qquad \Phi(0)=I.\]

For continuous, or bounded piecewise-continuous, \(A\), repeated integration gives the time-ordered expansion

\[\begin{aligned} \Phi(t)=I&+\int_0^t A(u)\,du\\ &+\int_{0<v<u<t} A(u)A(v)\,dv\,du+\cdots. \end{aligned}\]

If \(\|A(t)\|_{\mathrm{op}}\le M\), the term with \(j\) matrices has norm at most \((Mt)^j/j!\), so the series converges uniformly on bounded time intervals. Later matrices multiply on the left. In general, \(\Phi(t)\ne\exp(\int_0^tA(u)\,du)\): stretching and then rotating is not the same as rotating and then stretching.

Pairwise commutation, \(A(u)A(v)=A(v)A(u)\), is sufficient for the exponential-of-the-integral formula: the integrand in each term is then symmetric, so its ordered integral equals \(1/j!\) times \((\int_0^tA(u)\,du)^j\). Without commutation, the Magnus expansion organizes the correction terms using commutators [9].

Invertibility and Liouville’s formula

Solve \(\dot\Psi=-\Psi A\), \(\Psi(0)=I\). Differentiating \(\Psi\Phi\) gives zero, so \(\Psi=\Phi^{-1}\). Jacobi's determinant identity then yields Liouville's formula

\[\det\Phi(t)=\exp\left(\int_0^t\operatorname{tr}A(u)\,du\right)>0.\]

Thus a finite-time linear ODE never collapses dimension and preserves orientation. By contrast, an Euler block can be singular if \(I+hA_\ell\) is singular. This distinction will reappear at the saddles in the final experiment. \(\square\)

The training equations simplify too. With the Frobenius inner product on matrices,

\[\begin{aligned} -\partial_t p^i&=A(t)^\top p^i,\\ G_A(t)&=\frac1N\sum_i p^i(t)x^i(t)^\top+\lambda A(t),\\ \partial_s A_s(t)&=-G_{A_s}(t). \end{aligned}\]

For the discrete network, replace the outer product by \(p^i_{\ell+1}(x^i_\ell)^\top\) and multiply by \(h\) to obtain the ordinary Euclidean gradient. Even though the network is linear in its input, its dependence on all the matrices is nonlinear.

Numerical experimentation: linear ODE flows

The figure follows a circle of initial conditions over \(0\le t\le2\). Choose a constant matrix, a rotating strain, or two shears applied in different orders. The solid teal curve is the image of the unit circle; the dashed curve shows the prediction \(\exp(\int_0^t A)\). They agree in the constant case and can differ when matrices at different times do not commute.

The constant and switched-shear modes use exact formulas. The rotating-strain mode uses fourth-order Runge–Kutta and reports a step-halving discrepancy; the determinant can also be checked against Liouville's formula. The sliders change a prescribed coefficient path—they do not train a neural network.

Shared weights and the matrix exponential

If all layers share one matrix \(A\), then the endpoint maps become

\[F_L(A)=(I+A/L)^L, \qquad F_\infty(A)=e^A.\]

This is a much smaller model than freely choosing \(A(t)\). Weight tying also changes the gradient: the contributions of all layers must be summed. That sum is already of order one; the extra factor \(L\) used for independent layer parameters should not be applied again to the tied parameter. For a perturbation \(H\), with \(B=I+A/L\),

\[DF_L(A)[H]=\frac1L\sum_{j=0}^{L-1}B^{L-1-j}HB^j,\] \[D\exp(A)[H]=\int_0^1 e^{(1-u)A}He^{uA}\,du.\]

When \(A\) and \(H\) commute, the second expression simplifies to \(e^AH\). Let us choose a commuting two-dimensional family where the whole landscape can be drawn.

A two-dimensional commuting model

Identify the plane with \(\mathbb C\) and set

\[A(z)=\begin{pmatrix}x&-y\\y&x\end{pmatrix}, \qquad z=x+iy.\]

This matrix acts as multiplication by \(z\). All such matrices commute. Their exponential acts as multiplication by \(e^z\): the real part produces dilation, the imaginary part rotation. Take the target map to be \(-I\), a half-turn. The squared fitting loss, with no weight penalty, is

\[\begin{aligned} f_L(z)&=\tfrac12\|F_L(A(z))+I\|_F^2 =\left|(1+z/L)^L+1\right|^2,\\ f_\infty(z)&=\left|e^z+1\right|^2 =e^{2x}+2e^x\cos y+1. \end{aligned}\]

The Frobenius loss sums squared errors on the two coordinate inputs. This is a loss for real two-dimensional linear networks, not a requirement to implement complex-valued neural layers.

Theorem 1 (all critical points in this slice). For every finite integer \(L\ge1\), the global minima have value zero and are exactly

\[z_k=L\left(e^{i(2k+1)\pi/L}-1\right), \qquad k=0,\ldots,L-1.\]

For \(L=1\) there is no other critical point. For \(L\ge2\), the only other critical point is \(z=-L\), of value one. It is an ordinary saddle when \(L=2\), and a degenerate saddle with zero Hessian when \(L\ge3\). At infinite depth the critical points are exactly

\[z_k=(2k+1)\pi i,\qquad k\in\mathbb Z,\]

all global minima; there are no saddles. In the Euclidean coordinates \((x,y)\), the Hessian at each minimum is \(2I_2\). In particular, these landscapes have no nonglobal local minima.

Proof of Theorem 1: zeros and critical points of a holomorphic function

Write \(f=|g|^2\). In the real coordinates \((x,y)\), its first variation is

\[Df(z)[\delta]=2\operatorname{Re}\left(\overline{g(z)}\,g'(z)\delta\right).\]

This vanishes for every \(\delta\in\mathbb C\) exactly when \(g(z)=0\) or \(g'(z)=0\). Here

\[g_L(z)=(1+z/L)^L+1, \qquad g_L'(z)=(1+z/L)^{L-1}.\]

The zeros of \(g_L\) give the listed minima; \(g_L'\) only vanishes at \(-L\) when \(L\ge2\). At a minimum, \(|g_L'|=1\), so \(f_L(z_k+\delta)=|\delta|^2+O(|\delta|^3)\), giving Hessian \(2I_2\).

At the remaining critical point, put \(z=-L+\delta\). The exact expansion is

\[f_L(-L+\delta)=1+2\operatorname{Re}\bigl((\delta/L)^L\bigr)+|\delta/L|^{2L}.\]

Along rays with \(\cos(L\arg\delta)=+1\) the value increases; along rays with \(\cos(L\arg\delta)=-1\) it decreases for sufficiently small nonzero \(\delta\). Thus this is a saddle. When \(L=2\), the leading term is \((\operatorname{Re}\delta)^2/2-(\operatorname{Im}\delta)^2/2\), so the Hessian has eigenvalues \(1,-1\). For \(L\ge3\), the first nonconstant term has degree \(L\), and the Hessian vanishes.

Finally, \(g_\infty=e^z+1\) has derivative \(e^z\), which never vanishes. Its zeros are exactly the odd multiples of \(\pi i\), where again \(|g_\infty'|=1\). This exhausts the critical points. \(\square\)

Depth dependence of critical points and gradient flow

For a fixed integer \(k\), the corresponding finite-depth minimum satisfies

\[z_k=i(2k+1)\pi-\frac{(2k+1)^2\pi^2}{2L}+O(L^{-2}).\]

Thus minima near the origin approach the imaginary axis, while the saddle \(-L\) moves off to the left. For fixed \(L\ge2\), the gradient near that saddle satisfies

\[\|\nabla f_L(-L+\delta)\| =2\left|1+(\delta/L)^L\right|\left|\delta/L\right|^{L-1} \sim 2\left|\delta/L\right|^{L-1} \quad(\delta\to0).\]

This follows directly from \(\|\nabla|g|^2\|=2|g||g'|\). The gradient vanishes to increasingly high order: a saddle can be very flat without being locally convex. Here every residual block collapses to zero at \(-L\). This point lies outside any fixed bounded-parameter region in the ODE limit.

Nor does the absence of nonglobal local minima guarantee that gradient descent finds a minimum. For Euclidean gradient flow in \((x,y)\) at infinite depth, the real axis \(y=0\) is invariant. There,

\[\frac{dx}{ds}=-2e^x(e^x+1),\qquad \frac{dy}{ds}=0.\]

Indeed, \(u=e^{-x}\) satisfies \(du/ds=2+2/u\), so \(u(s)\sim2s\). Consequently \(x(s)=-\log(2s)+o(1)\) and \(f_\infty(x(s))-1\sim1/s\): the loss decreases toward one, not zero. The algorithm escapes along an asymptotically flat direction rather than converging to a finite bad critical point. Each finite-depth loss grows to infinity as \(|z|\to\infty\), but this boundedness of sublevel sets is lost in the limit. The exceptional real-axis trajectory is a useful warning, not a description of all initializations.

These conclusions concern the full two-real-parameter commuting slice, without regularization. They are not a landscape theorem for arbitrary matrix parameterizations or nonlinear ResNets. Even restricting to the real line changes the answer: for even \(L\), \(z=-L\) minimizes the real-line loss at value one, although it is a saddle in the plane. Odd finite depths, in contrast, have the real zero-loss solution \(z=-2L\). Rotation is necessary to reach the half-turn at even depth and in the ODE limit; the odd-depth real solution escapes to infinity as \(L\) grows.

Numerical experimentation: finite- and infinite-depth loss landscapes

Change depth without moving the camera. The colored curves are the exact loss levels \(0.1,0.5,1,2,5,10,25\); level one is dashed. For finite \(L\ge2\), zoom out to find the saddle at \(-L\), or zoom in to see its alternating sectors. “Infinity” evaluates the exponential itself, not a very large power. The visible-minimum counter counts all minima inside the plotting window.

Analytic parametrization of loss contours

A uniform grid can miss nearly all the structure when \(f_L-1\) is tiny. Instead, trace the level \(f_L=c\) in the image plane:

\[w(\alpha)=-1+\sqrt c\,e^{i\alpha}, \qquad (1+z/L)^L=w(\alpha).\]

For \(w\ne0\), taking every complex root gives

\[z=L\left[|w|^{1/L} e^{i(\arg w+2\pi k)/L}-1\right], \qquad k=0,\ldots,L-1.\]

At finite depth, \(0<c<1\) gives \(L\) separate loops around the minima: the circle in the \(w\)-plane does not surround zero. For \(c>1\), it winds once around zero, so the root sheets join into one closed curve. At \(c=1\), the \(L\) loops meet at the unique preimage of \(w=0\), namely \(-L\), a saddle when \(L\ge2\). The renderer samples the branches all the way to this point explicitly. At infinity, the corresponding formula is

\[z=\log|w|+i(\arg w+2\pi k),\qquad k\in\mathbb Z,\]

and \(w=0\) has no finite preimage. The missing saddle is replaced by leftward asymptotes. The renderer samples these analytic curves adaptively in screen coordinates and draws only the relevant portion of the infinite family. The paths are numerical polylines, but their vertices come from the exact level-set parametrization, not from interpolating rounded loss values on a grid.

The ODE formulation describes forward propagation and gives the adjoint training equations. Even for linear blocks, time ordering matters. The two-dimensional commuting example also shows that a convergent forward model can still have flat saddles or gradient-flow trajectories that fail to reach a minimum.

Bibliography

  1. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. CVPR, 2016. Paper.
  2. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. ECCV, 2016. Paper.
  3. Weinan E. A Proposal on Machine Learning via Dynamical Systems. Communications in Mathematics and Statistics 5, 1–11, 2017. Paper.
  4. Eldad Haber and Lars Ruthotto. Stable Architectures for Deep Neural Networks. Inverse Problems 34, 014004, 2018. Paper.
  5. Qianxiao Li, Long Chen, Cheng Tai, and Weinan E. Maximum Principle Based Algorithms for Deep Learning. Journal of Machine Learning Research 18(165), 1–29, 2018. Paper.
  6. Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. NeurIPS, 2018. Paper.
  7. Matthew Thorpe and Yves van Gennip. Deep Limits of Residual Neural Networks. Research in the Mathematical Sciences 10, 6, 2023. Paper · Correction.
  8. Pierre Marion, Adeline Fermanian, Gérard Biau, and Jean-Philippe Vert. Scaling ResNets in the Large-depth Regime. Journal of Machine Learning Research 26(56), 1–48, 2025. Paper.
  9. Sergio Blanes, Fernando Casas, José A. Oteo, and José Ros. The Magnus Expansion and Some of Its Applications. Physics Reports 470, 151–238, 2009. Paper.
  10. Michael Sander, Pierre Ablin, and Gabriel Peyré. Do Residual Neural Networks Discretize Neural Ordinary Differential Equations? NeurIPS 35, 36520–36532, 2022. Paper.
  11. Pierre Marion, Yu-Han Wu, Michael Sander, and Gérard Biau. Implicit Regularization of Deep Residual Networks towards Neural ODEs. ICLR, 2024. Paper.

How these posts and numerical experiments were prepared: AI disclosure.

Subscribe with an RSS reader: RSS Atom

How to subscribe

RSS and Atom let a reader app collect new posts for you; they are not email subscriptions. Opening these links in Safari may display plain text instead of a subscription form.

  1. Open an RSS reader, such as NetNewsWire on Mac.
  2. Choose Add feed or Subscribe in your reader (NetNewsWire calls it New Web Feed), then paste the RSS feed URL below. You can also use the Atom link above.

To copy manually, select the URL and press Command+C on Mac or Ctrl+C on Windows/Linux.

Back to the blog · Gabriel Peyré