A matrix can have many factorizations. Multiplying one factor by a scalar and dividing the other by the same scalar changes neither the matrix nor its predictions. Euclidean gradient descent acts on the factors, so two representations of the same model can start moving at different speeds.
In our work with Sibylle Marcotte and Rémi Gribonval [1], we study when training admits an intrinsic description, directly in terms of the represented model. This requires retaining some conserved quantities from initialization. We can calculate them explicitly for a two-factor linear model.
I owe my understanding of these dynamics to Sibylle Marcotte. In particular, the results on relaxed balancedness presented below are due to her, as developed in our joint work with Rémi Gribonval.
Closed dynamics for the represented model
Let a smooth parameterization \(z=\phi(\theta)\) represent the model, with loss \(F(z)\). The chain rule gives
\[\dot\theta=-D\phi(\theta)^*\nabla F(z),\qquad \dot z=-K(\theta)\nabla F(z),\] \[K(\theta)=D\phi(\theta)D\phi(\theta)^*.\]The star denotes the Euclidean adjoint. The operator \(K\) is a mobility: it converts a loss gradient into a model velocity. A closed equation for \(z\) requires recovering \(K(\theta)\) from \(z\), possibly together with quantities fixed at initialization. The equation must hold for all smooth losses \(F\), rather than describe only one particular trajectory.
The distinction matters for implicit bias. A reparameterized Euclidean flow generally is not Euclidean gradient flow in model space; the induced mobility records the geometry of the parameterization [2], [3]. We should not call \(K\) itself a metric: where invertible on the relevant tangent space, it is the inverse metric.
Scalar factorization and conserved imbalance
Start with \(z=ab\), where \(a,b\in\mathbb R\), and train \(F(ab)\). Then
\[\dot a=-bF'(z),\qquad \dot b=-aF'(z),\qquad \dot z=-(a^2+b^2)F'(z).\]The same product need not have the same mobility: \((a,b)=(1,1)\) and \((2,1/2)\) both represent \(z=1\), but their mobilities are \(2\) and \(17/4\). What is missing is the conserved imbalance
\[\lambda=a^2-b^2,\qquad \dot\lambda=0.\]Indeed, \((a^2+b^2)^2=(a^2-b^2)^2+4a^2b^2\). We therefore obtain an exact, closed equation:
\[\dot z=-\sqrt{\lambda^2+4z^2}\,F'(z).\]All dependence on the factors has collapsed into one number \(\lambda\). For the square loss \(F(z)=\tfrac12(z-z_*)^2\), the represented model relaxes toward its target with a state-dependent speed. Balanced initialization means \(\lambda=0\), with mobility \(2|z|\); a nonzero imbalance removes its degeneracy at \(z=0\). This last observation concerns the scalar equation; it does not remove the rank assumptions in the matrix theorem below.
Balanced matrix factorizations
Let \(A\in\mathbb R^{n\times r}\), \(B\in\mathbb R^{m\times r}\), and \(Z=AB^\top\). Write \(G=\nabla F(Z)\), using the Frobenius inner product. Factor gradient flow reads
\[\dot A=-GB,\qquad \dot B=-G^\top A, \qquad \dot Z=-AA^\top G-G BB^\top.\]Proposition 1 (conservation of latent imbalance). Along this flow,
\[A(t)^\top A(t)-B(t)^\top B(t) =A(0)^\top A(0)-B(0)^\top B(0).\]Proof: the two Gram matrices have the same derivative
Subtract the two identities. The loss and the dimensions play no further role. \(\square\)
This is the familiar balancedness conservation law, studied in deep homogeneous models [4] and in the systematic analysis of conservation laws [5]. If the two latent Gram matrices are equal initially, they stay equal. In that case,
\[(AA^\top)^2=A(B^\top B)A^\top=ZZ^\top.\]Since \(AA^\top\) is positive semidefinite, it must be the unique positive semidefinite square root of \(ZZ^\top\). The same argument applies to \(BB^\top\).
Theorem 1 (balanced intrinsic flow). Under balanced initialization \(A(0)^\top A(0)=B(0)^\top B(0)\), the product satisfies
\[\dot Z=-(ZZ^\top)^{1/2}\nabla F(Z) -\nabla F(Z)(Z^\top Z)^{1/2}.\]The preceding square-root calculation is the proof. There is no need to choose singular vectors, which may fail to be unique. On a fixed-rank stratum this is the familiar intrinsic geometry of balanced linear-network training [2], [3].
Relaxed balancedness with scalar imbalance
Balance is not the only useful level set. Assume that both factors have full column rank \(r\leq\min(n,m)\) and that
\[A^\top A-B^\top B=\lambda I_r.\]This is relaxed balancedness in [1]: an exact isotropic imbalance, not an arbitrary small departure from balance. Proposition 1 makes \(\lambda\) constant. Let \(P_Z\) be the orthogonal projector onto the column space of \(Z\), and \(P_{Z^\top}\) the projector onto its row space.
Theorem 2 (relaxed-balanced intrinsic flow). For as long as both factors have full column rank, define
\[\mathcal A_\lambda(Z) =\tfrac12 P_Z\bigl(\lambda I_n+\sqrt{\lambda^2I_n+4ZZ^\top}\bigr),\] \[\mathcal B_\lambda(Z) =\tfrac12 P_{Z^\top}\bigl(-\lambda I_m+\sqrt{\lambda^2I_m+4Z^\top Z}\bigr).\]Then \(AA^\top=\mathcal A_\lambda(Z)\), \(BB^\top=\mathcal B_\lambda(Z)\), and
\[\dot Z=-\mathcal A_\lambda(Z)\nabla F(Z) -\nabla F(Z)\mathcal B_\lambda(Z).\]Proof: solve a quadratic equation on the correct subspace
Put \(X=AA^\top\). Relaxed balancedness gives
\[ZZ^\top=A(A^\top A-\lambda I_r)A^\top=X^2-\lambda X.\]Thus \(X\) commutes with \(ZZ^\top\). Full column rank implies \(\operatorname{range}(Z)=\operatorname{range}(A)\). On this range, \(ZZ^\top\) is positive definite and the only positive root of the scalar equation \(x^2-\lambda x=s\), for \(s>0\), is
\[x=\frac{\lambda+\sqrt{\lambda^2+4s}}2.\]Functional calculus gives the displayed formula for \(X\) on the range of \(Z\). On its orthogonal complement, \(X=0\): this is exactly what \(P_Z\) enforces. Likewise, \((BB^\top)^2+\lambda BB^\top=Z^\top Z\) gives the second formula. Substitute into the product equation. \(\square\)
Numerical experimentation
Keep the initial product fixed, then move between imbalance levels. The left panel shows the factors traveling along the invariant curves \(a^2-b^2=\lambda\); the right shows their products. Equal initial predictions do not imply equal training trajectories.
Dependence on initialization
Once the conserved initialization label is fixed, the model has an autonomous equation. Its dynamics still depend on initialization: different \(\lambda\)'s produce different mobilities. Nor do arbitrary matrix-valued imbalances automatically lead to such a simple closed product flow. Our paper analyzes precisely when conservation laws provide enough information, with further results for deeper linear networks and path-lifted ReLU networks [1].
Factorization changes the geometry of the flow and can thereby change optimization and implicit regularization [6]. The companion post One low-rank matrix, many optimization speeds asks a complementary question: among representations of the same matrix, which are best conditioned? The geometry of sign gradient descent then changes the optimizer rather than the representation.
Bibliography
- Sibylle Marcotte, Gabriel Peyré, and Rémi Gribonval. Intrinsic training dynamics of deep neural networks. ICLR, 2026.
- Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: implicit acceleration by overparameterization. ICML, 2018.
- Bubacarr Bah, Holger Rauhut, Ulrich Terstiege, and Michael Westdickenberg. Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers. Information and Inference, 11(1), 307–353, 2022.
- Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homogeneous models: layers are automatically balanced. NeurIPS, 2018.
- Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Abide by the law and follow the flow: conservation laws for gradient flows. NeurIPS, 2023.
- Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. NeurIPS, 2017.