Newton methods include several classes, including * discrete* Newton,
* quasi* Newton (QN) (also termed * variable metric*), and
* truncated* Newton (TN). Historically, the
memory
requirements and
computation associated with solving a linear system
directly have restricted Newton methods only: (1) to small problems,
(2) to problems with special sparsity patterns, or (3) near a solution,
after a gradient method has been applied. Fortunately, advances in
computing technology and software are making the Newton approach
feasible for a wide range of problems. Particularly,
with advances in automatic differentiation [35,53],
the appeal of these methods
should increase further.
Extensive treatments of Newton methods can be found in
the literature ([16,23,30,45], for example)
and only general concepts will be outlined here.

For large-scale applications,
essentially two specific classes are emerging as the
most powerful techniques:
* limited-memory*
quasi-Newton (LMQN) and truncated Newton methods.
LMQN methods attempt to combine the modest storage and computational
requirements of CG methods with the superlinear convergence properties of
standard (i.e., full memory) QN methods. Similarly,
TN algorithms attempt to retain
the rapid quadratic convergence rate of classic Newton methods while
making computational requirements feasible for large-scale functions.