A matrix can have many factorizations. Multiplying one factor by a scalar and dividing the other by the same scalar changes neither the matrix nor its predictions. But Euclidean gradient descent sees the factors, not just their product: two representations of the same model can start moving at different speeds.
Can we nevertheless describe training directly in the space of represented models? In our work with Sibylle Marcotte and Rémi Gribonval [1], we study when such an intrinsic dynamics exists. The key is to keep the right conserved information from initialization. A two-factor linear model makes the idea completely explicit.
The question: can the model evolve on its own?
Let a smooth parameterization \(z=\phi(\theta)\) represent the model, with loss \(F(z)\). The chain rule gives
\[ \dot\theta=-D\phi(\theta)^*\nabla F(z),\qquad \dot z=-K(\theta)\nabla F(z), \] \[ K(\theta)=D\phi(\theta)D\phi(\theta)^*. \]The star denotes the Euclidean adjoint. The operator \(K\) is a mobility: it converts a loss gradient into a model velocity. A closed equation for \(z\) requires recovering \(K(\theta)\) from \(z\), possibly together with quantities fixed at initialization. This should work for all smooth losses \(F\), not just by fitting a vector field along one particular trajectory.
The distinction matters for implicit bias. A reparameterized Euclidean flow generally is not Euclidean gradient flow in model space; the induced mobility records the geometry of the parameterization [2], [3]. We should not call \(K\) itself a metric: where invertible on the relevant tangent space, it is the inverse metric.
A scalar model reveals the missing variable
Start with \(z=ab\), where \(a,b\in\mathbb R\), and train \(F(ab)\). Then
\[ \dot a=-bF'(z),\qquad \dot b=-aF'(z),\qquad \dot z=-(a^2+b^2)F'(z). \]The same product need not have the same mobility: \((a,b)=(1,1)\) and \((2,1/2)\) both represent \(z=1\), but their mobilities are \(2\) and \(17/4\). What is missing is the conserved imbalance
\[ \lambda=a^2-b^2,\qquad \dot\lambda=0. \]Indeed, \((a^2+b^2)^2=(a^2-b^2)^2+4a^2b^2\). We therefore obtain an exact, closed equation:
\[ \dot z=-\sqrt{\lambda^2+4z^2}\,F'(z). \]All dependence on the factors has collapsed into one number \(\lambda\). For the square loss \(F(z)=\tfrac12(z-z_*)^2\), the represented model relaxes toward its target with a state-dependent speed. Balanced initialization means \(\lambda=0\), with mobility \(2|z|\); a nonzero imbalance removes its degeneracy at \(z=0\). This last observation concerns the scalar equation; it does not remove the rank assumptions in the matrix theorem below.
Balanced matrices: the same mechanism
Let \(A\in\mathbb R^{n\times r}\), \(B\in\mathbb R^{m\times r}\), and \(Z=AB^\top\). Write \(G=\nabla F(Z)\), using the Frobenius inner product. Factor gradient flow reads
\[ \dot A=-GB,\qquad \dot B=-G^\top A, \qquad \dot Z=-AA^\top G-G BB^\top. \]Proposition 1 (conservation of latent imbalance). Along this flow,
\[ A(t)^\top A(t)-B(t)^\top B(t) =A(0)^\top A(0)-B(0)^\top B(0). \]Proof: the two Gram matrices have the same derivative
Subtract the two identities. The loss and the dimensions play no further role. \(\square\)
This is the familiar balancedness conservation law, studied in deep homogeneous models [4] and in the systematic analysis of conservation laws [5]. If the two latent Gram matrices are equal initially, they stay equal. In that case,
\[ (AA^\top)^2=A(B^\top B)A^\top=ZZ^\top. \]Since \(AA^\top\) is positive semidefinite, it must be the unique positive semidefinite square root of \(ZZ^\top\). The same argument applies to \(BB^\top\).
Theorem 1 (balanced intrinsic flow). Under balanced initialization \(A(0)^\top A(0)=B(0)^\top B(0)\), the product satisfies
\[ \dot Z=-(ZZ^\top)^{1/2}\nabla F(Z) -\nabla F(Z)(Z^\top Z)^{1/2}. \]The preceding square-root calculation is the proof. There is no need to choose singular vectors, which may fail to be unique. On a fixed-rank stratum this is the familiar intrinsic geometry of balanced linear-network training [2], [3].
Relaxed balancedness: retain one scalar
Balance is not the only useful level set. Assume that both factors have full column rank \(r\leq\min(n,m)\) and that
\[ A^\top A-B^\top B=\lambda I_r. \]This is relaxed balancedness in [1]: an exact isotropic imbalance, not an arbitrary small departure from balance. Proposition 1 makes \(\lambda\) constant. Let \(P_Z\) be the orthogonal projector onto the column space of \(Z\), and \(P_{Z^\top}\) the projector onto its row space.
Theorem 2 (relaxed-balanced intrinsic flow). For as long as both factors have full column rank, define
\[ \mathcal A_\lambda(Z) =\tfrac12 P_Z\bigl(\lambda I_n+\sqrt{\lambda^2I_n+4ZZ^\top}\bigr), \] \[ \mathcal B_\lambda(Z) =\tfrac12 P_{Z^\top}\bigl(-\lambda I_m+\sqrt{\lambda^2I_m+4Z^\top Z}\bigr). \]Then \(AA^\top=\mathcal A_\lambda(Z)\), \(BB^\top=\mathcal B_\lambda(Z)\), and
\[ \dot Z=-\mathcal A_\lambda(Z)\nabla F(Z) -\nabla F(Z)\mathcal B_\lambda(Z). \]Proof: solve a quadratic equation on the correct subspace
Put \(X=AA^\top\). Relaxed balancedness gives
\[ ZZ^\top=A(A^\top A-\lambda I_r)A^\top=X^2-\lambda X. \]Thus \(X\) commutes with \(ZZ^\top\). Full column rank implies \(\operatorname{range}(Z)=\operatorname{range}(A)\). On this range, \(ZZ^\top\) is positive definite and the only positive root of the scalar equation \(x^2-\lambda x=s\), for \(s>0\), is
\[ x=\frac{\lambda+\sqrt{\lambda^2+4s}}2. \]Functional calculus gives the displayed formula for \(X\) on the range of \(Z\). On its orthogonal complement, \(X=0\): this is exactly what \(P_Z\) enforces. Likewise, \((BB^\top)^2+\lambda BB^\top=Z^\top Z\) gives the second formula. Substitute into the product equation. \(\square\)
Numerical experimentation
Keep the initial product fixed, then move between imbalance levels. The left panel shows the factors traveling along the invariant curves \(a^2-b^2=\lambda\); the right shows their products. Equal initial predictions do not imply equal training trajectories.
What is—and is not—intrinsic
The model does evolve autonomously once the appropriate initialization label is fixed. But it is not independent of initialization: different \(\lambda\)'s produce different mobilities. Nor do arbitrary matrix-valued imbalances automatically lead to such a simple closed product flow. Our paper analyzes precisely when conservation laws provide enough information, with further results for deeper linear networks and path-lifted ReLU networks [1].
The point is not that factorization merely complicates a familiar flow. It changes its geometry, and can thereby change optimization and implicit regularization [6]. The companion post One low-rank matrix, many optimization speeds asks a complementary question: among representations of the same matrix, which are best conditioned? The geometry of sign gradient descent then changes the optimizer rather than the representation.
Bibliography
- Sibylle Marcotte, Gabriel Peyré, and Rémi Gribonval. Intrinsic training dynamics of deep neural networks. ICLR, 2026.
- Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: implicit acceleration by overparameterization. ICML, 2018.
- Bubacarr Bah, Holger Rauhut, Ulrich Terstiege, and Michael Westdickenberg. Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers. Information and Inference, 11(1), 307–353, 2022.
- Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homogeneous models: layers are automatically balanced. NeurIPS, 2018.
- Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Abide by the law and follow the flow: conservation laws for gradient flows. NeurIPS, 2023.
- Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. NeurIPS, 2017.