 |
Center for Nonlinear Studies |
Fitting model parameters to experimental data by least squares minimization is an ubiquitous problem in science that can be notoriously difficult for nonlinear models with many parameters. The problem, however, has an elegant geometric interpretation: the set of all possible model parameters induce a manifold in data space, with the best fit being the point on the manifold closest to the data. We discover that a wide variety of nonlinear fits have model manifolds that share common, universal features: they all have hierarchy of widths (spanning many orders of magnitude) and relatively small extrinsic curvatures. We explain both widths and curvatures using theorems from interpolation theory. The corresponding cost landscape is a hierarchy of narrow canyons and broad flat plateaus. Most fitting difficulties are understood to be algorithms stalling as they approach the boundaries on the manifold, i.e. being lost on the plateaus. Algorithms additionally become sluggish when the coordinates on the manifold are poorly suited to describing model behavior, i.e. when the canyons are curved. We use our geometrical insights to improve the standard fitting method (the Levenberg-Marquardt algorithm) by adding a geodesic acceleration term to the usual step, and find significant increases in both efficiency and success rates at finding best fits.
Host: Misha Chertkov, chertkov@lanl.gov, 665-8119