-
descent methods are used for minimizing or maximizing an
objective function where there is no constraints
-
the general technique is to produce a sequence of improved
approximations leading to the minimum or maximum point
according to the following scheme:
-
start with an initial trial point X1
where X denotes a vector of the independent variables
-
find a suitable direction Si which points to
the general direction of the maximum or minimum
-
find an suitable step length li for movement
along the direction Si
-
obtain the new approximation Xi+1 =
Xi + li Si
-
if f(X) is the function that needs to be minimized or maximized
then finding li boils down to finding the value of
li which minimizes f(Xi+1) =
f(Xi + li Si) for fixed
values of Xi and Si
-
here S is the gradient vector (derivative) of the function f(X) and by
moving along the direction pointed by the gradient one can
reach the maximum. The negative of S denotes the direction of
steepest descent