It is reasonable to choose a search vector that will be a descent direction; that is, a direction leading to function reduction. A descent direction is defined as one along which the directional derivative is negative:
When we write the approximation
we see that the negativity of the right-hand side guarantees that a lower function value can be found along for a sufficiently small .
Different methods are distinguished by their choice of search directions. Algorithms can be classified into nonderivative, gradient, and second-derivative (Newton) methods depending on the technique used to determine in Algorithm 2.1. These classes will be discussed in turn beginning in Section 2.3.