L-BFGS

BFGS is a quasi-Newton method. It builds an approximation $H_k\approx ({\nabla }^2\Phi {)}^{-1}$ of the inverse Hessian from gradient differences and steps along $p_k=-H_k\nabla \Phi ({\sigma }_k)$. With $s_k={\sigma }_{k+1}-{\sigma }_k$, $y_k=\nabla {\Phi }_{k+1}-\nabla {\Phi }_k$ and ${\rho }_k=1/(y_k^{\top }{s}_k)$:

$$H_{k+1}=(I-{\rho }_k{s}_k{y}_k^{\top })H_k(I-{\rho }_k{y}_k{s}_k^{\top })+{\rho }_k{s}_k{s}_k^{\top }.$$

The update keeps $H$ positive definite as long as $y_k^{\top }{s}_k>0$, which a Wolfe Line Search guarantees.

Limited memory (L-BFGS). Dense $H_k$ is impossible for $10^4$–$10^6$ unknowns. L-BFGS stores only the last $m$ pairs $(s_k,y_k)$, typically $m=5$–$20$, and applies $H_k$ with the two-loop recursion in $\mathcal{O}(mn)$ operations. The initial matrix $H_0={\gamma }_{k}I$ with ${\gamma }_k=\frac{s^{\top }y}{y^{\top }y}$ sets the scale. A problem-adapted $H_0$, such as the inverse Mass Matrix, makes the method mesh-independent (see Gradient Representation and the Riesz Map).

For EIT. It needs only objective values and gradients, one state and one adjoint solve per pattern and evaluation (see Adjoint State Method). Curvature is learned along the way. It usually converges much faster than steepest descent, with superlinear local convergence. For bound constraints use L-BFGS-B.

In ModularEIT.jl: LBFGS.

References

1. D. C. Liu, J. Nocedal (1989). On the limited memory BFGS method for large scale optimization. Math. Program. 45, 503–528. doi:10.1007/BF01589116
2. J. Nocedal, S. J. Wright (2006). Numerical Optimization, 2nd ed., Ch. 6–7. Springer. doi:10.1007/978-0-387-40065-5