Optimization: Problems, Vector Norms, and Convex Sets

Introduction

Presentation of optimization issues and addition of some of the most basic knowledge reserves

The questions in this article can also be addressedMathematical analysis: the theory of limits and continuity、Higher algebra: the basis of the meta-mathematicsHow the concept of a relatively close read together is developed in different contexts.

On the issue of optimization

The most important aspect of mathematical modelling is that it is often combined with mathematical and computer numerical methods. The general optimization problem is reflected in The polar question of the target function under binding conditions And the tools in math analysis are often unmanageable. At the very beginning of the optimal problem, we often see some extremes of the indifferent constraints, and then, of course, we'll have more.

Basic concepts

Common concepts

The equation and the array of the different kinds of constraints is the problem. Available Set The best in the world. It's the largest or the smallest of the target functions, and if this is the only point, then it's called Strictly global and optimal Local best solution The maximum or minimum value under a particular neighbourhood, not the whole set of possibilities. The best solution is often difficult to study in the whole world, and many of the methods behind are only local best solutions. $z=min{f(x_{1},x_{2})}$ This two-dimensional optimization problem is often a polar issue of curves. Theoretically: The best solution to any global event is closed. - Yeah. Theoretically: If the target function, the equation, the differential constraint, are all continuous functions, the feasible field is closed. - Yeah.

Knowledge supplement

Vector Paradigm

Definition: A measurement structure is the extension of the concept of modeling. $||x||$

Nature of the model:

- It's a good time.>0$
- That's right. $||cx||=c||x||$
Triangular Instinct $||x+y|| \ge ||x||+||y||$

Different vector-based definitions He's just a measurement structure, not the only one.

EuroPerformance $US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$$$US$$US$US$$$$US$$$$$$$$$$$$$US$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$2 = \sqrt{\sum{i=1}^{n} x_i^2}$$
1 standard $US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$$$US$$US$US$$$$US$$$$$$$$$$$$$US$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$1 = \sum{i=1}^{n} |x_i|$$
Infinity $US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$$$US$$US$US$$$$US$$$$$$$$$$$$$US$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$${\infty} = \maxI'm not gonna get a chance to get a chance to get a chance to get a chance to get a chance to get a chance to get a chance to get a chance to work. The European paradigm that we use most often in analyticals, he's called the 2th.

Vector sequence condensation

Concealed by standard $||x^{b}-x^{k}||$ Limit to 0
Concentrate by coordinates.
Two contractions are essentially equal.

Hessian Matrix

The gradient of the multiple function is a vector, and the weights of the vector are also a multifunctional function. In Optimizing Theory$\nabla$ It usually means the gradient count. He's linear. A few special examples. $$\bigtriangledown (b^{T}x)=b$$ $$\bigtriangledown (x^{T}x)=2x$$ $$\bigtriangledown (x^{T}Ax)=2Ax$$

Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum, Crum,

It's a fundamental concept that is important to both linear and non-linear planning, and it's a very necessary intellectual complement, but it's not very deep.

Crumb

Basic concepts

Definitions:$C$It's a geometry.~~x,y\in C~~ if~~\lambda x+(1-\lambda)y\in C~~\lambda\in[0,1] $C is a concussion The corresponding concepts are condensed. For the crumb verification, use definition directly to complete It's easy to verify the following propositions, which involve a collection of condensed figures, which are real numbers. $$\beta S_{1}={\beta x|x\in S_{1} }是凸集$$ $$S_{1}\cap S_{2}是凸集$$ $$S_{1}+S_{2}={x^{1}+x^{2}|x^{1}\in S_{1}~~~x^{2\in}S_{2}}$$ $$S_{1}-S_{2}={x^{1}-x^{2}|x^{1}\in S_{1}~~~x^{2\in}S_{2}}$$

Cream and Multi-Face

Polars and polars

Definition: If S is a non-empty condensation,$x\in S$ If x cannot be the condensed combination of two different points in S, called x the polarity of the condensed S. The polygons have polar points at their top, and every one of them in the circle is polar. Inference: For a climax, any single one of these can be a polar condensed condensation, and no one can be able to be able to be able to assemble. Definitions: Establishment$S$Yes.$R^{n}$Up the closed cam $d$It's not zero.$S$♪ Every one of them ♪$x$ There's all the rays. $${x+\lambda d|~\lambda \ge0}\in S$$ Name$d$Yes.$S$If one direction cannot be the sum of the other two, then this direction.$d$Yes.$S$♪ The polar direction ♪ It's obvious that only the assembly of the unbounded can have the concept of direction, so only the unbounded can have the extreme direction. Inference: Any direction can be a positive linear combination of extreme directions

Crumb Separation Theorem

The intuitive meaning of the amplification of the separation theorem is that under very weak conditions, two interminglings can always be separated by a super-platform, i.e., for super-platform. $p^{T}x=a$ Both assembly points are satisfied. $p^{T}x_{1}\ge a$ and$p^{T}x_{2}\le a$ This is what it's called.$H$ Separate. Two sets.

Combling

Definitions: For definitions in the condensation C Functions on $f(x)$ If for $$$$$$$$ x, y\inC\forall \lambda \in[0,1]$有$f(\lambda x+(1-\lambda)y)\le \lambda f(x)+(1-\lambda)f(y)$ 则称这个函数是凸函数如果$x\ne y$ is called strict condensation (so-called condensed) when you do not take the equivalent And the following reasoning and theorem are also defined to prove it. Definition: If the numeric number above is in reverse, it is called the dent. (This corresponds to the one dollar permutation definition) Definitions:$f(x)$is a convex function$-f(x)$It's a dent. Two definitions of equal value Definitions: taken$\lambda=1/2$ It's called a mid-point. Inference: Linear functions are both condensed and dented Theoretically: Two amphibious functions are combined with a condensed function. Inferences:$f(x)$It's a convex.$\Longrightarrow$ $\Omega_{c}={x|x\in \Omega,f(x)<It's a condensed condensation Theorem: Consequence of cones on the condensation Theoretically: Definition in the climax$C$Micro functions on$f(x)$It's a convex.$\Leftrightarrow$ $\forall x,y\in C,f(y)\ge f(x)+\nabla f(x)^{T}(y-x)$ If you're demanding strict, the equals are deleted. Theoretically: from a one-dollar camcorder extension definition is opening Set$C$Micro functions on$f(x)$It's a convex. $\Leftrightarrow$ $f(x)$Hessian Matrix is semi-positive.$\nabla^{2}f(x)\ge 0$It's just a matter of changing from a condition of being a mere necessity to a condition of being a sufficient one.

Cam Planning

Definitions: both target and binding functions are the planning questions of the condensed function called the condensed planning Inference: The linear planning problem is Cam. Inference: The viable set of cam is the cam, the best is cam, the best part of the place is the best in the world. Theoretically: For cam planning, if the target function is strict and the best solver exists, the best solver exists and the only one is The only solution is that multiple points achieve the same optimal function, otherwise the best concept cannot be discussed. Theorem: set to $x{}$是凸规划$(P)$的可行解则其是最优解的充要条件是 $x^{}$ 是规划$min_{x\in S }\nabla f(x^{*})^{T}x$的最优解其中S是$(P)Approbable Fields of $$

Basic nature of linear planning

Linear Programing, his binding and target functions are linear, which is a simpler, more basic type of optimisation, and we're going to study the general solution to linear planning, and it's important to give some basic elements of linear planning and special solutions before we do it.

Standard form of linear planning

Standard form

Linear planning has so-called standard forms, binding functions can be a mixture of equations and variations, target functions can be maximized, but linear planning has standard forms, which are useful for the description of the solutions that follow. Theoretically, all linear planning can be translated into the following forms, known as standard forms of linear planning. The standard form of linear planning may be in the form of an equation:$$\begin{aligned} \min_{x_1,x_2,\cdots,x_n} \quad & c_1 x_1+c_2 x_2+\cdots+c_n x_n \ \text{s.t.} \quad & a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n= b_1 \ & a_{21}x_1+a_{22}x_2+\cdots+a_{2n}x_n= b_2 \ & \cdots \ & a_{m1}x_1+a_{m2}x_2+\cdots+a_{mn}x_n= b_m \ & x 1, x 2, \cdots, x n\geq0 \end{aligned} $ I'm not sure.$x_i$ For the first time $i$ A decision variable,$c_i$ For the first time $i$ The coefficient of the individual decision variable,$a_{ij}$ For the first time $i$ of the $j$ The coefficient of the individual decision variable,$b_i$ For the first time $i$ right-end constant of a condition. If expressed in matrix $$\begin{aligned} \min_{x} \quad & c^T x \ \text{s.t.} \quad & A x = b \ & You're not gonna get a chance to get a job. I'm not sure.$x$ Yes. $n$ - The vector.$c$ Yes. $n$ - The vector.$A$ Yes. $m\times n$ The matrix,$b$ Yes. $m$ . Vector. Here. $\max$ Means maximize target function $c^T x$，$\text{s.t.}$ Expressing binding conditions. First line is the target function, second line is the binding condition

Number of variables$n$Called$LP$The dimensions of the problem. The number of equations.$m$Called$LP$Step of the problem>I'm gonna need a little help. Some of the corresponding matrix of unrelated constraints are called the foundation of linear planning. The equations that the matrix corresponds to are decomposing as the fundamentals of the matrix. Break The conditions of restraint permit what is called feasible Break The best way to meet our target function is to be called the best solution. Obviously, yes.$m$Step$n$V-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D-D--D----$LP$Questions, at most.$C_{n}^{m}$Basic viable solutions, with each matrix format corresponding to one basic viable solution

To standard form

First of all, we need to deal with the difference between the min max of the target function, which is actually, when the max of the target function is the opposite of the min, which is very easy to simplify. For the question of the equation below, the equation needs to be introduced into the equation, as follows: $$a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n \ge b_1$$ Introduce loose variables into $$a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n -x_{n+1}= b_1~~~~x_{n+1}\ge 0$$ Same thing. $$a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n\le b_1$$ Other Organiser $$a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n+x_{n+1}= b_1~~~~~x_{n+1}\ge 0$$ Please note that every new variant has to introduce new relaxing variables, not repeat, and the relaxing variable must be larger than zero, adjust its own positive and negative numbers.

Treatment of Free Variables

Our standard form is to require that all variables are positive and that free variables are not allowed to appear. Here is an example of how to process them, the core of which is to eliminate them by deciphering an equation. $$00begin{aligned}\min x 1,x 2,\cdots,x 3}\quad & x_1+3 x_2+4 x_3 \ \text{s.t.} \quad & x_1+2x_2+x_3= 5 \ & 2x_1+3x_2+x_3= 6 \ & x_2,x_3\geq 0 \end{aligned}$$ 目前这个问题中$x_{1}$是自由变量，明显这不符合我们的需求，所以我们解第一个等式方程得到 $$x_{1}=5-2x_{2}-x{3}$$ 代入前面的表达式得到 $$\begin{aligned} \min_{x_1,x_2,\cdots,x_3} \quad & 5-2 x_2- x_3 \ \text{s.t.} \quad & x_2+x_3= 4 \ & x 2, x 3\geq 0 \end{aligned} $ And then it's all done. It's all over.

Figure

It's easier to figure out the simplest linear method. We draw the graphics of the binding conditions and the linear functions to be sought, and by moving the line, we can find the maximum value. The obvious pattern is that it can only deal with two dimensions. Inference: For a two-dimensional linear planning problem, the best solution must be found at the convex of the convex. And if the two vertexes are the best, then it means that the best is not the only one, but a line. And when the assembly is infinite, the best solution may not exist, and it needs to be taken into account.

Theories of linear planning

The basic rationale for linear planning is two core theorems, which are the basis for the solution to the problem of linear planning behind us. $$定理：矢量x是凸集Ax=b的极点的充要条件是x是Ax=b的一个基本可行解$$ $$定理：对于一个标准的LP问题。如果存在可行解，就一定存在基本可行解，如果存在最优可行解，就一定存在最优的基本可行解$$ Two of theorems can tell us that the best solution to the LP problem is to study the basic viable solution, the polar point of the feasible collection, which gives us the most basic and central elements of the solution, and a review of some of the elements of the diagram. $$推论：只要可行集非空，至少有一个极点$$ $$推论：只要有限的最优解存在，那一定在一个极点上$$ $$推论：极点的数量至多为有限个$$

Basic simple methods

Here we'll present the solution to one of the core LP questions. According to the basic theorem, we can find the best solution when we look at all the poles, but it is very unrealistic to have a super-large LP problem, so we need to give a simple approach; He is a conversion method, from one basic viable to another, and ensures that the target function is reduced, and that, over time, the optimal basic viable solution is found with fewer than a few times. We'll introduce the three inverted lines of thought, and then we'll combine them for a simple table exercise.

Basic decomposition

For one.$LP$The problem is as follows: $$00begin{aligned}\quad & c^T x \ \text{s.t.} \quad & A x = b \ & x \geq 0 \end{aligned}$$ 由于我们的原始问题是通过引入松弛变量实现的标准化，所以我们认为系数矩阵中一定存在一个标准阵$E$ 即如下的形式 $$\left{\begin{array}{ll} x_{1}+ & +y_{1, m+1} x_{m+1}+\cdots+y_{1 n} x_{11}=y_{10} \ x_{2}+\cdots \ & Other Organiser I'm sorry, I'm sorry. It's easy to find a basic solution at this point. How do we find another basic solution? Suppose we want to remove the basic variable xp and introduce the new basic variable xq, and our operation is to... $p$Okay. $x_{q}$coefficient to 1 (multiplication constant) Other rows $x_{q}$coefficients to 0 (plus minus)$p$Lines)

Assurance of feasibility

Study the feasibility of ensuring that the process of conversion is resolved.

Decrease in the number of determinations and target functions

Study the reduction of the target function that is guaranteed to be solved during conversion

Use of simple tables

Let's give you an example of that.

Simple methods for improvement

Big M.

To deal with the absence of a matrix, we introduced the Big M method, which is the following. We're introducing artificial variables into the original linear planning problem.$y=(y_{1},y_{1}...y_{m})$ tectonic linear planning issues $$00begin{aligned}\quad & c^T x+ME^{T}y\ \text{s.t.} \quad & A x+y = b \ & You're not gonna get a chance to get a job. At this point, there is a unit matrix that can be solved using a simple method. Theorem: set (x^)},y^{})$ 是修正问题的最优解那么如果$y^{}$为0 $x^{It's the best solution to the original problem Otherwise, there's no solution to the problem. Ry. It's pointless to make M a certain number.

Two-stage approach

Or is it the original problem that we introduce artificial variables?$y=(y_{1},y_{1}...y_{m})$ The question of the tectonic amendment is as follows: $$00begin{aligned}\min & \sum y_{i}\ \text{s.t.} \quad & A x+y = b \ & x \geq 0 ~y \geq 0\end{aligned}$$ Theorem: set (x^)},y^{})$ 是修正问题的最优解那么如果$y^{}$为0 $x^{It's the best solution to the original problem Otherwise, there's no solution to the problem. Ry.

Degradation and recycling

If a min=0 appears in the calculation of the entry base variable, the new resolution corresponds to the target function of the old resolution, resulting in an iterative cycle that goes beyond We don't give you the way to deal with degradation and cycling, just to avoid problems like this as much as possible.

There are more than one dollar.<We pick the smallest one as the base variable.
If you have multiple off-base variables, choose$min~r_{k}$That one.

One-dimensional search.

The construction rationale for optimizing the problem

Here is a description of the overall rationale for optimizing the problem. Take the unbounded question of miniaturization$min(f(x))$ For a given initial value$x^{0}$ We have an iterative process that's as follows:

I'm gonna get it by certain rules.$x_{k}$Falling direction$d_{k}$
Make sure you have a long walk by certain rules. {k}$ 一般是$min[f(x^{k}+\lambda{k}d_{k})]$
You!$x^{k+1}=x^{k}+\lambda_{k}d_{k}$
To determine, according to certain rules, whether or not it is necessary to end the iterative, circular or output results You can get a sequence according to the above method.$x^{k}$ His limit is the tiny problem of miniaturization. Points If there are very small values common to multiple initial points, it's a global contraction or local.

Condensity analysis

The compulsive analysis of algorithms is a complex exercise, and many algorithms are not sure whether they are constricting, but they are not used in a way that will delay us. Considering the efficiency of implementation is a more important issue in the design of algorithms, and some discussion of conservativity follows.

For a sequence that is condensed by a standard number$x^{k}$ If real number exists$\alpha$ and constants $k$ Satisfied $lim k\rightarrow\infty}\flex()}x{}\right|}{\left|x^{k}-x^{}\right|^{\alpha}}=q$$

$\alpha=1~q>It's called linear compression speed.
$1<\alpha<2~~q>0~~or~~\alpha=1~~q = zero dollars called ultralinear contraction
$\alpha=2$ It's called a second-stage containment speed.

Based on the condensity analysis, they give the usual iterative termination conditions, which are used to optimize the above-mentioned problems.

$||x^{k+1}-x^{k}||<\varepsilon$
$||\bigtriangledown f(x^{k})||<\varepsilon$

Accurate one-dimensional search.

In the optimal question construction in the front, step long$\lambda$It is based on a one-dimensional, tiny-value problem, which is essentially the optimization of the one-dimensional basis, and we are here to study the calculation of this problem, i.e., a one-dimensional search; we will only introduce some of the methods, because they are not exhaustive;

It's an analytical solution.

For a one-dimensional problem, we can study it directly by looking for guidance. $f'♪ (x) = zero is the point of the extreme, just solve the corresponding value. ♪

Success - Failure Law

A one-dimensional unbounded problem.$min(f(x))$ For a given initial value$x^{0}$ The calculation is as follows:

Give initial points$x^{0}$ Search step £h>0$ 精度$\varepsilon$
Calculate$x^{1}=x^{0}+h;f_{1}=f(x^{1})$
If $f s<Other Organiser $x^{0}=x^{1},f_{0}=f_{1},h=2h$
On the contrary, the search failed if $US$|<\varepsilon$ 找到极小，搜索结束否则缩小步长后退搜索$Other Organiser This is the whole iterative process of success. No title to search for spaces$\varepsilon$ Our goal is to find the best possible possible compartment, not the exact value, and the theory of it is perfectly consistent.

Act No. 0.618

A means of integrating through the principle of separation of gold. Consider the following one-dimensional miniaturization $min(f(x))stI'm not sure I'm gonna be able to do this. We need to know in advance. $\varepsilon,\alpha=0.618$

Calculating $lambda 1}a1}(1-\alpha)(b a )~~\mu_{1}=a_{1}+\alpha (b_{1}-a_{1})$ $f(\lambda_{1})~~f(\mu_{1})$
If $b k}-a k}<\varepsilon$ 迭代结束最优解为$(b_{k}+a_{k})/2$ 如果$f(\lambda_{1})>Turn 3 or turn 4
$a_{k+1}=\lambda_{k},b_{k+1}=b_{k}$ And then, in this range, we start again from 1st.
$a_{k+1}=a_{k},b_{k+1}=\mu_{k}$ And then, in this range, we start again from 1st.
Angular Cursor The number of overlaps required by the 0.618 law is often higher because of his very slow pace of contraction.

Diphthesis

The dichotomy here is perfectly consistent with the dichotomy of rooting, and we use the mediaorical and guidanceal reasoning to determine the exact location of root; we simply need to stop repeating it, and we need to look at the cut-off conditions, which are usually based on $US$xx1}1}{}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}$}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}<\varepsilon This is a very similar core idea to the Newton traverse, which is to turn the smallest value problem of the original function into a zero point problem of the conductive function and then solve it using a rooting method.

Newton Theory

His core idea is to use the second-order Taylor expansion to approximate the original function, to study the position of the tiny values with the analytical nature of the approximate function, which is the following infra. For a given initial value$x^{0}$ We're going to do the iterative process. $US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$US$$US$US$US$US$$$$US$$$$$$$$$US$$$$$US$$$$$$$$US$$$$$$$$$US$$$$$$'}(x^{k})/f^{''(x^k) $ Until then,'}(x^{k}|<\varepsilon The advantage of this method is that it's very fast to absorb.

Blanket method

Using a second-stage function other than Taylor to achieve a similarity, i.e., a multi-value-plug-in approach, we choose three-plug-in to construct a second-order function. We need initial plug-in point $x .~~x_{1}~~x_{2}$ 一般默认$The factor of the second-order function can be obtained by demanaging the interpolation equation that we construct in the middle of the equation. $$\bar{x}=1/2[x_{1}+x_{0}-b_{1}/a_{2}]$$ of which $$(f f)/(x x )=b (f {2}-f )/ (x 2}=b 2}(b 2(2)-b (x ) / (x}2}) =a (2} $ Just understand. Better remember the formula.

Unprecision 1-D search

In some cases, an imprecision one-dimensional search can act as a better acceleration of the contraction, and an accurate one-dimensional search can be a little too difficult to achieve, so an imprecision one-dimensional search is now used very widely. It's a way out of the way.

Unbound optimisation method

Extreme condition for unbound optimization

In general, unbounded optimization is achieved through a series of 1-dimensional searches, and the choice of this series of 1-dimensional searches is a matter that we need to consider; Theorem: The curve drops fastest in the direction of the negative gradient, and the gradient at the very small point is zero. Theorem: The best local conditions for filling need to be added to the Hessian matrix. Theorem: For the amphibious function, the best solution in the whole area is a zero gradient. With these theories, we know that the gradient is at the heart of the study of the problem of the non-binding polarity.

Maximum Drop

The core of the most rapid drop is the principle of the fastest drop in the negative gradient; then the one-dimensional search determines the length of each step, and the end is the end of the cut-off, followed by a detailed calculation step.

Give initial points$x^{0}$ Precision$\varepsilon$ You!$k=0$
Calculate$d^{k}=-\bigtriangledown f(x^{k})$ If you want to go to the hospital,<\varepsilon$搜索停止，返回现在的$x^{k}$作为$x^{*}$
From$x^{k}$ Let's go. Let's do a one-dimensional search.$min{f(x^{k}+\lambda d^{k}}$
Found it.$\lambda$Find a new one.$x^{k+1}$ Next iterative In manual computing, a one-dimensional search often uses the simplest analytical properties to achieve search, not those complex search algorithms, which are more suitable for computer use. The algorithm is excellent.

There's no need for a start point. The front is fast.
The iterative speed is slow at the point of most excellent resolution, unstable for disturbances, the rate of containment is influenced by the scale of the variable, leading to a problem of initial point selection or not, and bad initial point selection leads to complex computation processes
For determining the best in the global context, there are often multiple points for the best in the local context, and if consistent, the best in the global context is considered.
To avoid a decline in the serene state, we're using modified search directions.$d^{k}=x^{k}-x^{k-2}$ It's probably understandable. He can avoid a fall in the serene serene.

Newton Act

Newton uses the double approximation to achieve the search orientation, as follows:

Give initial points$x^{0}$ Precision$\varepsilon$ You!$k=0$
Calculate$g^{k}=\bigtriangledown f(x^{k})$If $g^k}<\varepsilon$搜索停止,返回现在的$x^{k}$作为$x^{*}$
Calculate$d^{k}=-[\bigtriangledown ^{2} f(x^{k})]^{-1}g^{k}=-H_{k}^{-1}g^{k}$ As a direction of decline
Make a one-dimensional search.$min{f(x^{k}+\lambda d^{k}}$
Found it.$\lambda$Find a new one.$x^{k+1}$ Next iterative $H^{-1}$ It's a countervailing matrix. To ensure that the matrix is in place.$Hk$and$gk$We can multiply the matrix. We're gonna make a column vector for the gradient. The algorithm is excellent.

Local deflation speed is excellent, secondary termination, optimal for double contour.
Newton's direction is not necessarily down, is limited to the hessian matrix's heaeness, and requires a non-generic matrix, otherwise the directional calculation is difficult

Co-graduation

It's not easy to calculate the second-order Hessian matrix and his reverse matrix; the lateer-stage effects of the most rapid decline are not good, and we want to combine the two advantages, the co-graduation approach we're introducing here. Definitions: vectors$d_{1}~d_{2}$ About a matrix co-exist means$d_{1}^{T}Ad_{2}=0$ This matrix is a unit when it's down. $orord: A n*>0,d^d^n}is a set of A-Cyber vectors; f(x)=1/2x^T}Ax+b^T}+c needs to start from any initial point, and a precise one-dimensional search from each co-location direction can find the best solution, at most $n$ With the aberrations, the problem becomes the generation of a group of co-directions, which in fact is the best combination of functions, and we give the most classic co-radical algorithms directly below.

Give initial points$x^{1}$ You!$k=1$
Calculate$d^{k}=-g^{k}=-\bigtriangledown f(x^{k})$ Gradients small enough to terminate the iterative
One-dimensional search according to the direction of the negative gradient, giving a supporting formula$\lambda _{k}=\frac{(g^{k})^{T}g^{k}}{(d^{k})^{T}Ad^{k}}$ (analytical calculations also)
You! $x^{k+1}=x^{k}+\lambda_{k}d^{k};g^{k+1}=\bigtriangledown f(x^{k+1});\alpha_{k}=\frac{(g^{k+1})^{T}g^{k+1}}{(g^{k})^{T}g^{k}};d^{k+1}=-g^{k+1}+\alpha_{k}d^{k}$
The number of iteratives is the number of variables, counting the number of times from the initial point Subaru Please note that we need to ensure that the first negative gradient is used to calculate the direction of the construction. $A$It's the initial Hessian matrix.

The DFP method of variation (Newton Act)

Newton Equation

The newton method is an improvement on the newton method, and it's hoped to find an iterative way to replace the Hessian matrix. $$H_{k+1}g_{k}=s_{k}$$ It's called the Matrix.$\times$Current gradient = direction of decline.

Newton Act.

The idea of a newton law is to replace the Hessian matrix in the Newton approach with other means, which is a huge flaw in the newton law.

Give initial points$x^{0}$ Precision$\varepsilon$
You!$H_{1}=E$ Calculate$g^{1}=\bigtriangledown f(x^{1})$ If you're not gonna get it,<\varepsilon$搜索停止,返回现在的$x^{1}$作为$x^{*}$
You!$d_{k}=-H_{k}g^{k}$
One-dimensional search and find out.$\lambda_{k}$ Calculate$x^{k+1},g^{k+1}$
Repeat the operation and use the DFP to fix the formula to get new$H_{k}$ Until we find the best solution. $$DFP fixation formula: {H}{k+1}={H}{k}+\frac{\Delta x^{k}\left(\Delta x^{k}\right)^{T}}{\left(\Delta x^{k}\right)^{T} \Delta g^{k}}-\frac{{H}{k} \Delta g^{k}\left({H}{\kn\k}The difference between \delta g^ and k is $$$$$$

Banning optimization methods

Limitation polar conditions

A binding optimization should be expressed in the following form: the whole is divided into target functions, with varying degrees of binding, with equations bound by three parts. $$\left{array} \min {x \in \mathbb{R}^{n}} f(x) \ \text { s.t. } g{i}(x) \geq 0, i=1, \cdots, m, \ h_{j}(x)=0, j=1, \cdots, n . \end{array}\right.

Gather!

S=\left{\begin{array}{l|l} x \in \mathbb{R}^{n} & \begin{array}{l} g_{i}(x) \geq 0, i=1, \cdots, m, \ h_{j}(x)=0, j=1, \cdots, n . \end{array} \end{array}\right}

Available set for question (1). $

For the optimization of the existence of binding, it is likely that the location of the target function without binding is not feasible Set$S$It's not possible to do research directly using unbound methods. We need to adapt. Definition: Falling direction, target function falling as it moves in this direction Definition: a feasible direction, and as we move in this direction, we can guarantee that only a certain length will remain viable. Internal Definitions: Linearly feasible direction set of a point $US$(\,=),=left(=,{\)= The \neq 0 \matbb{r & \begin{array} The \nabra g (\), \geq0, i \i, \i, \i \ =0,j=1,\cdots, n. I'm sorry. I'm sorry, I'm sorry. Definitions: For a workable set$S$One of these, the differential binding conditions are divided into two states, and are met.$g(x_{i})=0$♪ When called positive binding>0$ 称为非积极约束，记$I = i = 0, i = 1, 2,... m}As a positive constraint indicator for a point Set $Kuhn-Tucker polar condition: Sets target and binding functions can be micro, vector collections of CQ {\bigtriangledown g (\) (\), \\bigtriangledown h {j} (\bar{x}} linearly irrelevant; there is a number w i} if \\bar{x} is the best solution in part~~♪ V j} Made~~\\bigtriangredownf (\)=sum w sigtriang (+bar)+sumv sum \bigtriangledownh} (\barx}); where w i}\g0~ ~~i \in I~~I is a binding set of indicators Point to make KT polar conditions valid$\bar{x}$Called$K-T$Points Theorem: the problem of condensed planning$K-T$The best part of the world.

$K-T$Validation and calculation of points

In fact, we are doing this with full definition, and we need to practice and understand in depth the process we need to calculate, and we need to handle a linear equation group that can get K-T point positions; likewise, we can do validation. It's still a bit difficult to understand the calculation. The only issue with equations is the LAGL's method of multiplying mathematically, so the KT method is to deal with the two constraints of the equation and combine them. First, there's only one kind of indifferent constraint.

Construct the LaGrand function $L(x,\lambda)=f(x)+\lambda g(x)$
Scrambling gradient equation $\nabla f (x^)})+\lambda \nabla g(x^{})=0$
Separate discussion$\lambda=0~and\nabla g(x^{*})=0$
Verify if the resolution obtained meets the gradient equation and $\lambda\ge 0~g(x^{*})\le0$ So we can know exactly which KT points we need to know. Consolidated
Construct the LaGrand function $L(x,\lambda)=f(x)+\sum\limits\lambda_{i} g_{i}(x)+\sum\limits\mu_{j}h_{j}(x)$
Scrambling for gradients $\nabla f(x)+\sum\limits\lambda_{i} \nabla g_{i}(x)+\sum\limits\mu_{j} \nabla h_{j}(x)=0$
Separate discussion$\lambda_{i}=0~and\nabla g_{i}(x^{*})=0$
We discuss each and every one of them on our own.$\lambda_{i}$ When it's zero, it's the same.$g_{i}(x)$Not zero, or it'll work.$g_{i}(x)$- It's not.
Verify if the resolution obtained meets the gradient equation and $\lambda i}\ge0~~g(x^{*})\le0~~h_{i}(x^{*})=0$ It's not easy to be wrong to be careful about the standard form of binding.

SUMT Extralegal

SUMT is the unbounded method of miniaturization. The idea is to construct a punitive function that allows our own variables to deviate from the feasible domain and then expand rapidly, and then change to unbounded optimization. Step one. Construct penalty function$p(x)$ Okay.$F(x,M)=f(x)+Mp(x)$ $p(x)$♪ Need to meet the continuum, constant, just$x\in S,p(x)=0$ Step two. Construct this$p(x)$It's easy to think about. We have the following idea. Equivalent Constraints$p(x)=h^{2}(x)$ Impansive constraints$p(x)=0 if x\in S else~ ~p(x)=g^{2}(x)$ For multiple constraints, all of them.$p(x)$It's good to add up. Step three. Solution Unbound Optimization If you need it, it's a different story.$M$ Use$M_{k+1}=M_{k}c;c\in[4,10]$ Initial$M$We need to specify that, in the computation of analysis, we need to remember that M is very big.

SUMT Intra-Mechanism

So the idea is to construct the wall function that allows us to rapidly increase the function after the variable is near the viable boundary, and to change it to an unbounded optimization. Step one. Construct penalty function$B(x)$ Okay.$F(x,r)=f(x)+rB(x)$ $B(x)$It's a constant, constant, and it's a time when B(x) is going to be endless. Step two. Construct this$B(x)$The item is easy to think of, which one is good, and which one is good, and the standard form is more than zero. $B(x)=\sum\limits g_{i}^{+}(x);g_{i}^{+}(x)=-\frac{1}{g_{i}(x)}or-ln(-g_{i}(x))$ Step three. Solution Unbound Optimization If you need it, it's a different story.$r$ Use$r_{k+1}=\frac{r_{k}}{c};c\in[4,10]$ Initial$r$We need to specify that, in the computation of analysis, we should remember that r is close to zero.

SUMT Mixing Method

Combined punishment and bumping walls can accelerate the iterative process.

Multi-purpose planning

The definition of multi-purpose planning is very clear and clear; we have a number of$min~max$function;

It is clear that if all target functions achieve optimal interpretation (absolutely optimal) at the same point, it is certainly good, but it is too ideal to achieve;

In fact, we tend to give the concept of effective solvency here; it means that there is no better solution than he (achieving one goal function would necessarily be another one that deviates from the best), the core point is that the size of the vector is not comparable here, and the final multi-target planning issue is called the question of choosing in a valid solve according to one's preferences, how to choose, without distinction of superiority or inferiority.

Our multi-purpose planning is a process of finding effective solutions, not the best solution, and the best solution is determined by preference;

Some of the classic solutions for effective solutions are described below, with the idea of reducing multiple goals to single targets;

Very small, very large.

$$h(x)=min/max~{h_{1}(x),h_{2}(x)...h_{n}(x)}$$ The disadvantage, only one effective solution, the resulting function lost its microlasticity

Linear weighting

$$h(x)=\sum \theta_{i}h_{i}(x)$$

Square weight

$$h(x)=\sum \theta_{i}(h_{i}(x)-h_{min}(x))~~~h_{min}(x)=min~{h_{1}(x),h_{2}(x)...h_{n}(x)}$$

Multiply

Another way to construct, to study later.

Law of constraint

Only one target function is retained, and the remaining target functions are converted into a solution to binding conditions