There are many optimization algorithms available to the computational scientist. Many methods are appropriate only for certain types of problems. Thus, it is important to be able to recognize the characteristics of a problem in order to identify an appropriate solution technique. Within each class of problems there are different minimization methods, varying in computational requirements, convergence properties, and so on. A discussion on the relative strengths and weaknesses of available techniques within a given class of problems will be the focus of the following sections. Optimization problems are classified according to the mathematical characteristics of the objective function, the constraints, and the control variables.
Probably the most important
characteristic is the nature of the objective function. A
function is linear if the
relationship between 
and the control variables is of the
form
where 
is a constant-valued
vector and 
is a constant; a function is quadratic if
where 
is a
constant-valued matrix. Special methods can
locate the optimal solution very efficiently for linear and quadratic
functions, for example.
There is a special
class of problems, examples of which are particularly common in the
fields of operations research and engineering design, in which the
task is to find the optimum permutation of some control variables.
These are known as combinatorial optimization problems. The most
famous example is the traveling salesman problem (TSP) in which the
shortest cyclical itinerary visiting 
cities is sought. The
solutions to such problems are usually represented as ordered lists
of integers (indicating, for example in the TSP, the cities in the
order they are to be visited), and they are, of course, constrained,
since not all integer lists represent valid solutions. These and
other classifications are summarized in Table 1.
Table 2 lists application examples
from the wide range of fields where optimization is employed and
gives their classifications under this taxonomy.
Section 2 details methods appropriate to unconstrained continuous univariate/multivariate problems, and Section 3 mentions methods appropriate to constrained continuous multivariate problems. Section 4 considers methods appropriate to (mixed) integer multivariate problems, and Section 5 discusses what to do if none of these methods succeed. In general, the optimization of the complex functions that occur in many practical applications is difficult. However, with persistence and resourcefulness solutions can often be obtained.
