Blog

The Stone–Weierstrass theorem is a cornerstone of approximation theory: it turns a few structural properties of a family of functions into the ability to approximate every continuous function. It explains the density of polynomials and Fourier modes, and provides a route to universality for neural networks—even two-layer networks. It also plays a central role in my recent work with Takashi Furuya and Maarten de Hoop on attention universality over an infinite-dimensional space of contexts [3].

The hypothesis that often looks most restrictive is stability under products. Multiplying two polynomials gives another polynomial, but multiplying two ReLU networks generally does not give a ReLU network. Fortunately, exact multiplication is unnecessary: it is enough that products belong to the uniform closure of the class. This elementary consequence of the usual theorem is worth keeping in mind.

Neural networks are flexible representations of functions. But what if the function must be convex? This constraint is especially natural in optimal transport. For probability measures with finite second moments and an absolutely continuous source, Brenier’s theorem identifies the quadratic optimal transport map as the gradient of a convex potential [1], [2]. Learning such a potential is therefore a way to learn a transport map while respecting its geometry.

Input-convex neural networks, or ICNNs, enforce convexity through their architecture [3]. A natural question follows: can they approximate every convex function? Yes, if depth is allowed to grow with the desired accuracy [4]. But in dimension two, no fixed depth suffices. We start with two explicit counterexamples on a square, for one and two hidden ReLU layers, then give a self-contained proof of the obstruction at every fixed depth.

Universality means reaching every prescribed uniform error by increasing the network’s size. Its failure is compatible with a good fit at a fixed tolerance: the obstruction can be small enough to escape a plot of the function values. The second example below illustrates this distinction particularly well.

If we translate two point clouds by the same vector, should their optimal transport plan simply translate along with them? For a cost such as the squared Euclidean distance, the answer is immediate: every pairwise displacement stays the same. But this is only one way for the optimal matching to be preserved.

The more interesting question is to characterize all costs whose optimal plans commute with translations. A cost may change without changing its minimizers: multiplying it by a positive constant or adding terms that depend on the source point alone or the target point alone has no effect on the optimizer. Under suitable nondegeneracy assumptions, these two freedoms give the complete answer.

While working with Stephen Zhang on the convergence of two-layer linear networks, we made use of a construction I find particularly appealing: mean-field conservation laws. These are functionals of the neuron distribution that remain constant during training. The same functional works for every number of neurons, and also for a continuous distribution describing an infinite-width network.

In our recent preprint, the relevant laws depend only on the neurons’ second moments. They turn a familiar conservation principle, balancedness, into a statement about a matrix whose size does not grow with the network. This gives a way to control the geometry of training and prove convergence.

How do you turn a transport plan into a map? This question appears as soon as we transport a small point cloud to a much larger one: the Kantorovich solution splits each source point’s mass among several destinations. Entropic regularization makes that splitting even more explicit.

The barycentric projection replaces those destinations by their conditional mean. For an arbitrary coupling, the resulting map need not be optimal. If the coupling solves quadratic Kantorovich OT or quadratic entropic OT, however, its barycentric projection is an unregularized quadratic Monge map—but to its own image measure, not generally to the original target. In the entropic case, this follows from a smooth convex log-sum-exp potential derived from the optimal dual potentials.

In an optimisation problem, we know the objective and look for the best decision. In inverse optimisation, we observe decisions and try to recover an objective that explains them. Inverse optimal transport (iOT) is one example: rather than computing a coupling from a cost, we learn a cost from an observed coupling.

A useful measure of disagreement is the gap loss: the cost of the observed decision minus the cost of the best decision. When the objective depends linearly on the parameter to learn, this loss is convex—even if the forward optimisation problem is not. This is closely connected to Fenchel–Young losses. The interactive experiment below explores the gap for transport between two mixtures of Gaussians.

I share many things with my colleague, friend, and office mate (well, not this year!) Olivier Cappé, including a love of Gaussians.

I especially love PDEs that preserve Gaussians—and, more specifically, Wasserstein gradient flows that do so. An infinite-dimensional evolution of a probability density then reduces exactly to the evolution of a mean vector and a covariance matrix.

This is the topic of the very last section of my new book Optimal Transport for Machine Learners: 16.5, “Flows over the Gaussian Manifold.”

J’ai été interviewé pour la chronique Nos vies numériques de France Inter du 10 septembre 2026, consacrée à l’intelligence artificielle et aux équations de Navier–Stokes. Mon intervention a été fortement raccourcie — ce que le format très court de la chronique peut expliquer — et l’extrait diffusé ne reflète pas l’ensemble de ma pensée.

Oui, les progrès récents de l’IA sont passionnants. On entrevoit de nouvelles façons de faire des mathématiques, qui ouvriront très vraisemblablement la voie à des avancées majeures. Mais cet enthousiasme ne peut pas être séparé des questions qu’ils soulèvent pour la recherche et l’éducation.

Convexity is not the only route to fast optimization. The Polyak–Łojasiewicz (P–L) condition asks for something more directly connected to gradient descent: away from the minimum value, the gradient must remain large enough.

For a differentiable function with minimum value \(f_\star\), the condition is

\[\frac12\|\nabla f(x)\|^2\geq \mu\bigl(f(x)-f_\star\bigr), \qquad \mu>0.\]

It rules out stationary points with a nonoptimal value, without requiring a convex landscape.

The Mathematical Nexus brings together 800 animated vignettes and 140 accompanying Python notebooks, encompassing most of the mathematical content I have shared on social media.

The idea is to give these short explorations a more permanent home: a place where one can move from a visual intuition to the mathematics and the code behind it, and follow connections between topics.

A positive Markov matrix repeatedly mixes probability vectors until they approach a single stationary distribution. Sinkhorn’s algorithm has a similar visual story: repeated application of a nonlinear map brings positive scaling vectors toward a distinguished ray.

The analogy is particularly striking on the simplex of probability vectors with three entries — a triangle. Linear mixing shrinks straight-sided triangles; Sinkhorn bends them as it contracts. The appropriate geometry for the nonlinear picture is projective.

Take Gaussian distributions \(\nu_i=\mathcal N(m_i,\Sigma_i)\) with positive definite covariances, and weights \(\lambda_i\geq0\) summing to one. Their barycenter for the objective

\[\min_\mu\sum_i\lambda_i\,\mathrm{KL}(\mu\mid\nu_i)\]

is again Gaussian. The striking point is that we average the precision matrices \(\Sigma_i^{-1}\), rather than the covariances themselves.

How does a long product of matrices change when each factor is perturbed? There is an exact answer that replaces one factor at a time:

\[A_1\cdots A_n-B_1\cdots B_n =\sum_{k=1}^n A_1\cdots A_{k-1}(A_k-B_k)B_{k+1}\cdots B_n.\]

This discrete Duhamel formula is simply a telescoping sum — but one whose ordering is indispensable when matrices do not commute.

For a convex function \(f:\mathbb R^d\to\mathbb R\cup\{+\infty\}\), write

\[f^*(y)=\sup_x\{\langle x,y\rangle-f(x)\}\]

for its Legendre transform. A beautiful consequence of Prékopa–Leindler is that the functional

\[\Phi(f)=-\log\int_{\mathbb R^d}e^{-f^*(y)}\,\mathrm dy\]

is convex, on a class of proper closed convex functions for which the integrals below are finite and strictly positive.

A symmetric matrix can be positive in two different senses: positive semidefinite, and nonnegative entry by entry. Requiring both gives the cone of doubly nonnegative matrices

\[\mathcal D_n=\{X\in\mathbb R^{n\times n}:X=X^\top,\ X\succeq0,\ X_{ij}\geq0\}.\]

A seemingly similar condition is to ask for a Gram factorization with nonnegative vectors. This defines the cone of completely positive matrices. The two cones agree up to dimension four — but not in dimension five.

The alpha version of my new book, Optimal Transport for Machine Learners, is out. In particular, there is an online version with interactive figures, so that the mathematical exposition can be explored alongside the examples.

This is an early version: the text and its illustrations will continue to evolve. My aim is to make the geometry and computation of optimal transport accessible to readers interested in machine learning, with the mathematics and its visual intuition close together.

A centered Gaussian can be transformed into three Gaussian clouds in many ways. Two particularly interesting constructions are the deterministic diffusion map, obtained by reversing a score-based probability-flow ODE, and the optimal transport map, which minimizes the total squared displacement.

Both produce the same target distribution in the continuous model. But matching distributions does not mean matching individual particles. Coloring each particle by its destination makes this distinction visible.

Given a way to measure a positive semidefinite matrix, a natural way to measure a general matrix is to apply it to its Gram matrix:

\[N(X)=\sqrt{\gamma(X^\top X)}.\]

When does this construction give a norm? The key condition is monotonicity in the positive semidefinite order: increasing a covariance should not decrease its size.

It is intuitively clear that straighter trajectories should be easier to integrate numerically. This intuition is especially appealing in flow matching, where one would like to generate samples with as few velocity evaluations as possible.

I wanted a precise estimate that exhibits this dependence directly. Rewriting the usual stability argument behind Cauchy–Lipschitz gives one: Euler’s global error is controlled by the material acceleration along the exact trajectory, multiplied by a stability factor.

In my paper Muon Dynamics as a Spectral Wasserstein Flow, I connect a matrix optimizer to optimal transport by changing how a distribution’s motion is measured.

For positive numbers, solving \(uyu=x\) is easy: \(u=\sqrt{x/y}\). What remains true when the variables are matrices, which need not commute?

If \(X\) and \(Y\) are symmetric positive definite, there is still exactly one symmetric positive definite solution of \(UYU=X\). More surprisingly, for fixed \(Y\), the correspondence between \(U\) and \(X\) is a global smooth change of coordinates on the positive definite cone.

Fun fact: a linear operator on matrices may mix all their entries, but preserving the rank of every matrix leaves remarkably little freedom. It must be a change of basis on the left and on the right.

There is just one additional possibility for square matrices: transposition, followed by those same changes of basis. The proof comes from understanding the linear subspaces made entirely of matrices of rank at most one.

A book I am delighted to recommend: Tout comprendre (ou presque) sur l’intelligence artificielle, by Olivier Cappé and Claire Marc, forthcoming from CNRS Éditions on 10 April 2025.

I have the pleasure of sharing my office with Olivier, so I was able to witness the creation of this fantastic book. Seeing the finished result is therefore particularly special.