Low-rank adaptation, or LoRA, trains a small matrix correction in factored form instead of updating every weight of a large model [1]. If this correction is \(AB\), replacing \((A,B)\) by \((sA,s^{-1}B)\) leaves it unchanged. The model is identical. Its Euclidean optimization geometry is not.
Our work with Valérie Castin, Kimia Nadjahi, and Pierre Ablin [2] studies this redundancy through local conditioning and proposes a balancing method. Here is a particularly clean result: for exact low-rank matrix recovery, every balanced factorization has the best possible positive Hessian condition number. The proof needs only a Jacobian, a Sylvester operator, and two singular-value inequalities.
Remove symmetry directions before measuring conditioning
Fix a matrix \(Y\in\mathbb R^{n\times m}\) of rank \(r<\min(n,m)\), and consider
\[ f(A,B)=\tfrac12\|AB-Y\|_F^2, \qquad A\in\mathbb R^{n\times r},\quad B\in\mathbb R^{r\times m}. \]We examine a minimizer satisfying \(AB=Y\); both factors necessarily have rank \(r\). For every invertible \(R\in\mathbb R^{r\times r}\), the pair \((AR,R^{-1}B)\) is another minimizer. Thus the Hessian has zero eigenvalues: moving along this factorization orbit changes the parameters without changing their product.
The relevant Euclidean condition number uses the positive spectrum, not a ratio with a zero denominator:
\[ \kappa_+(H)=\frac{\lambda_{\max}(H)}{\lambda_{\min}^+(H)}, \qquad H=\nabla^2 f(A,B). \]This is a local quantity at an exact solution. It controls the quadratic behavior transverse to the minimizer manifold; it is not a statement that the nonconvex loss is globally strongly convex. Quotient and manifold viewpoints provide a broader language for handling these symmetries [3].
Diagonalize the smaller operator
The differential of \((A,B)\mapsto AB\) is
\[ J(\Delta A,\Delta B)=\Delta A B+A\Delta B. \]At an exact fit the residual vanishes, so the Hessian is simply \(H=J^*J\). Instead of writing its large block matrix, use the fact that \(J^*J\) and \(JJ^*\) have the same positive eigenvalues. The latter acts on a matrix \(E\) as
\[ JJ^*(E)=AA^\top E+E B^\top B. \]Left and right multiplication are diagonal in eigenvector bases of the two Gram matrices. Their eigenvalues therefore add.
Theorem 1 (exact positive condition number). If \(AB=Y\) and \(r<\min(n,m)\), then
\[ \kappa_+(H)= \frac{\sigma_1(A)^2+\sigma_1(B)^2} {\min\{\sigma_r(A)^2,\sigma_r(B)^2\}}. \]Proof: add the left and right eigenvalues
Write \(\alpha_i=\sigma_i(A)^2\) and \(\beta_j=\sigma_j(B)^2\) for \(1\leq i,j\leq r\). The eigenvalues of \(AA^\top\) are the positive \(\alpha_i\)'s and \(n-r\) zeros; those of \(B^\top B\) are the positive \(\beta_j\)'s and \(m-r\) zeros. On each outer product of the corresponding eigenvectors, \(JJ^*\) acts by their sum.
Its positive eigenvalues are therefore
\[ \alpha_i+\beta_j,\qquad \alpha_i,\qquad \beta_j. \]The last two families occur because \(m-r>0\) and \(n-r>0\). The largest positive eigenvalue is \(\alpha_1+\beta_1\), and the smallest is \(\min(\alpha_r,\beta_r)\). These are also the extreme positive eigenvalues of \(H\). \(\square\)
Balance is optimally conditioned
A factorization is balanced when \(A^\top A=BB^\top\). It shares the scale of the product equally between the two factors.
Theorem 2 (balancing attains the optimum). Among all exact factorizations \(AB=Y\),
\[ \kappa_+(\nabla^2 f(A,B))\geq 2\frac{\sigma_1(Y)}{\sigma_r(Y)}. \]Every balanced factorization attains equality. In particular, if \(Y=P\Sigma Q^\top\) is a thin singular value decomposition,
\[ A=P\Sigma^{1/2},\qquad B=\Sigma^{1/2}Q^\top \]is optimally conditioned. This is the exact-recovery case of Proposition 2.4 in [2].
Proof: bound the numerator from below and the denominator from above
Submultiplicativity of the operator norm gives \(\sigma_1(Y)\leq\sigma_1(A)\sigma_1(B)\). Thus
\[ \sigma_1(A)^2+\sigma_1(B)^2 \geq2\sigma_1(A)\sigma_1(B)\geq2\sigma_1(Y). \]At the bottom of the nonzero spectrum, \(\sigma_r(Y)\geq\sigma_r(A)\sigma_r(B)\). To see this without a rectangular-matrix ambiguity, use thin QR decompositions \(A=U R_A\), \(B=R_B V^\top\), with \(U,V\) having orthonormal columns. The nonzero singular values of \(Y\) are those of the invertible square product \(R_A R_B\), whose smallest singular value is at least the product of the two smallest singular values. Consequently,
\[ \min\{\sigma_r(A)^2,\sigma_r(B)^2\} \leq\sigma_r(A)\sigma_r(B)\leq\sigma_r(Y). \]Insert these inequalities in Theorem 1. Finally, if \(C=A^\top A=BB^\top\), then \(A=U C^{1/2}\) and \(B=C^{1/2}V^\top\) for column-orthonormal \(U,V\). Hence \(Y=UCV^\top\), and the eigenvalues of \(C\) are exactly the positive singular values of \(Y\). Both inequalities become equalities. \(\square\)
The unavoidable part of conditioning is the singular-value spread of \(Y\). Imbalance adds an avoidable part. This distinguishes a difficulty of the represented problem from a difficulty introduced by its coordinates.
A one-parameter distortion of the same matrix
Keep the thin SVD of \(Y\) fixed and choose
\[ A_s=sP\Sigma^{1/2},\qquad B_s=s^{-1}\Sigma^{1/2}Q^\top, \qquad s>0. \]The product is always \(Y\), whereas Theorem 1 gives
\[ \kappa_+(H_s)=\frac{\sigma_1(Y)}{\sigma_r(Y)} \frac{s^2+s^{-2}}{\min(s^2,s^{-2})}. \]The minimum is at \(s=1\), and the degradation grows as \(s^4\) or \(s^{-4}\). Even a moderate mismatch of scales can have a substantial effect on the local condition number.
Numerical experimentation
Use \(Y=\operatorname{diag}(4,1,0)\), represented by two rank-two factors. The left panel is the exact Hessian condition number. The right runs actual factor gradient descent near each minimizer, starting with the same tiny perturbation norm and comparing it with the Hessian linearization.
Balanced, hyperbalanced, and genuinely nonlinear
There is still a rotational freedom in a balanced factorization. Calling it hyperbalanced means that the common Gram matrix is diagonal, with its entries ordered. This chooses a canonical form of balance. It is not needed to minimize the Euclidean quantity in Theorem 2: all balanced factorizations already do that. It can matter for coordinate-dependent optimizers, whose geometry is not invariant under latent rotations; see Which condition number does sign gradient descent see?
An SVD-based balancing map preserves the represented product. It should not be confused with a nearest-point Euclidean projection in factor space. The practical BaLoRA construction in [2] exploits small matrix decompositions to make this reparameterization efficient.
For a nonlinear least-squares model \(\tfrac12\|h(AB)-y\|^2\) at an interpolating solution, the paper gives conditioning bounds involving the Jacobian of \(h\). These require the corresponding rank and nondegeneracy hypotheses. If exact interpolation fails, residual terms enter the Hessian. Neither case is covered by simply reusing \(H=J^*J\) above, and neither turns the local calculation into a universal convergence theorem for fine-tuning.
Finally, gradient flow conserves latent imbalance, while finite-step gradient descent need not conserve it exactly [4]. The companion post on intrinsic factor dynamics explains the continuous-time reduction. Earlier analyses of factorization and implicit bias [5], as well as initialization-dependent LoRA dynamics [6], further illustrate why the representation at initialization is not an innocuous detail.
Bibliography
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: low-rank adaptation of large language models. ICLR, 2022.
- Valérie Castin, Kimia Nadjahi, Pierre Ablin, and Gabriel Peyré. Balanced LoRA: removing parameter invariance to accelerate convergence. ICML, 2026.
- Nicolas Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
- Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homogeneous models: layers are automatically balanced. NeurIPS, 2018.
- Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. NeurIPS, 2017.
- Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on LoRA finetuning dynamics. NeurIPS, 2024.