Markov Chains: Transition Probabilities, State Classification, and Stationary Distributions

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Markov Chain at Breaktime

And then we'll introduce a special series of random variables, which is characterized by the fact that the results are only now affecting the next results, and that the results before them have no impact on the next results; that's the Marcov chain. He's both mathematically easy to calculate and more in keeping with the randomness of reality, and is now widely used. Recent research has allowed the Malkov chain to play an important role in the Monte Carlo approach. We started working on it from the break-up.

Definition of the Markov chain at discrete time

Basic definitions

Definitions$X={X_n:n\geqslant0}$is defined in probability space (in the case of a "fault")$\Omega,\widehat{F},\mathbb{P})$random process on which the state space is a numerical assembly S if it is for any non-negative integer $n\geq0$ and$i_0,i_1,...,i_n,i_{n+1}\in S$ Yes. $$00\ &P(X_{n+1}=i_{n+1}|X_{0}=i_{0},X_{1}=i_{1},\cdots,\color{}{X_{n}=i_{n}}) \ &♪ I'm a little girl ♪ I'm sorry, I'm sorry. This random process is called a separate Markov chain of time.

The questions in this article can also be addressedRandom process basis: random process definition, digital characteristics and smooth process、Financial random analysis: financial derivatives, fork tree pricing and the theory of arbitrageHow the concept of a relatively close read together is developed in different contexts.

The definition of Markov Chain is the most essential feature we can callMarquisity.Or amnesia. Only now will the results affect the next results, and the previous results will not affect the next results. Conditional probability in expression $$P(X_{n+1}=i_{n+1}|X_{n}=i_{n})$$ Called from the state$i_{n}$To Status$i_{n+1}$And the probability of a step of diversion is recorded as$p_{i_{n}i_{n+1}}(n)$ Additional definitions: $$p_{ij}^{(k)}(n)\stackrel{{def}}{=}{P}(X_{n+k}=j\mid X_{n}=i)$$ From a state$i$Moving out. Experience.$k$- It's a transfer to Ta.$j$The probability of a$n$The moment.$k$Sub-transfer probability It's clear that the probability of a single transfer is...$k$Special case of the probability of a secondary transfer We're in a deal.$k$The matrix of secondary probability is presented below. $$\left.\matbf{P}}loft(n\right)=\left(\begin{matrix}}p}{ij}^{(k)}\left(n\right)\\end{matrix}\right.\right)$$ $k=1$的时候 一步转移概率矩阵为 $$\mathbf{P}^{(1)}(n)=(p{ij}^{(1)}(n))$$ 简记为 $\color{}{\mathbf{P}(n)=(p_{ij}(n))}$ 对于$k=0$的情况 我们约定 $$\left.p_{ij}^{(0)}(n)=\delta_{ij}=\left{\begin{matrix}{1,}&{i=j}\{0,}&{i\neq j}\\end{matrix}\right.\right.\quad(i,j\in S)$$ 此时转移概率矩阵是单位矩阵 不难验证 转移概率矩阵是随机矩阵 即 $$p_{ij}^{(k)}\left(n\right)\geqslant0,\quad\sum_{j\in S}p_{ij}^{\left(k\right)}\left(n\right)=1$$

Zhiji Markov chain.

In practical applications, we'll see a more specific nature of the Maast chain:One step to transfer probability and time$n$It's not about that.$$p_{ij}\left(n\right)=p_{ij}\left(n+1\right)=p_{ij}\left(n+2\right)=\cdots $$At this point, we call the chain "Cyber" "Cyber" or "Cyber" "Cyber" Chain We're looking at the chain after that, which is often a series of times, and it's a little complicated if we use different transfer probability matrices at every time. One intuitive way of expressing a probability of a single shift: state chart; it's just a matter of writing all the states, and then drawing the probability of a single move from each to the other.

Examples

We've given some examples of the Marcov chain and researched how to verify that a random process was Markov. Chain

Unlimited randomly swimming

Unlimited random motion on the pivot if$p$And the probability of one. $q$One less chance can shift the probability from definition to definition. $$p j==begin{cases}p,&\text{i=0,±1,±2,...,}j=i+1\q,&\text{i=0,±1,±2,...,}j=i-1\0,&|i-j|>1\end{cases}$$ 简单理解一下我们是怎么写出来的 从$i$状态一步转移到$j$状态的条件是两者得相邻 一侧是$p$ 另一侧是$q$ 如果不相邻 那概率就是0 这当然满足马氏性（从直观的理解上） 我们用定义也可以做到验证 我们尝试用定义验证他的马氏性 还是从变换的机理出发 把$X_n$变为一系列随机变量序列的和（里面的随机变量是广义的伯努利分布） $$\mathbf{X}{n}=\xi{0}+\xi_{1}+\cdots+\xi_{n}$$ 所以有 $$\begin{gathered} P(X_{n+1}=i_{n+1}\big|X_{0}=i_{0},X_{1}=i_{1},\cdots,\color{}{X_{n}=i_{n}}\big) \ =\frac{P(X_{0}=i_{0},X_{1}=i_{1},\cdots,X_{n}=i_{n},X_{n+1}=i_{n+1})}{P(X_{0}=i_{0},X_{1}=i_{1},\cdots,X_{n}=i_{n})} \ =\frac{P(\xi_{0}=i_{0},\xi_{1}=i_{1}-i_{0},\cdots,\xi_{n+1}=i_{n+1}-i_{n})}{P(\xi_{0}=i_{0},\xi_{1}=i_{1}-i_{0},\cdots,\xi_{n}=i_{n}-i_{n-1})} \end{gathered}$$ 根据独立性化简有 $$=P(\xi_{n+1}=i_{n+1}-i_{n})$$ 也就是 $$\begin{aligned}=\begin{cases}p,&i_{n+1}-i_n=1\1-p,&i_{n+1}-i_n=-1\0,&\text{Other}\end{cases}\end{aligned}$$ $$=P (x+n+n) $ So the definition also validates marzipanity, and the other questions of proof of marzipanity start from a similar point of view here.

There's a limit to randomly moving.

Set Particle Point on Line${0，1，···，a}$I'm gonna do random jiggling on every dot. I'm gonna do the following.

• Except...$0$and$a$Up the twilight $p$Shift right $q$Shift left $r$No change.
• Yes.$0$Move around$r_{0}$ No change. $p_0$ Shift right
• Yes.$1$Move around$r_{a}$ No change. $p_a$ Shift left Our rules are right about time.$n$No restrictions. All of them are Zidmar. Chain One step to transfer the probability matrix to $ \matbf{&p_0&0&0&\cdots&0&0&0\q&r&p&0&\cdots&0&0&0\0&q&r&p&\cdots&0&0&0\\vdots&\vdots&\vdots&\vdots&\cdots&\vdots&\vdots&\vdots\0&0&0&0&\cdots&q&r&p\0&0&0&0&\cdots&0&q_a&r_a\end{bmatrix}$$

Group growth

Each individual with a biological group produces a future by its own means during its lifetime, assuming that each individual is based on probability.$p_k$Generate$k$A generation, and there are $$p_k\geq0,\sum_kp_k=1$$ Use$X_n$Other Organiser$n$He's a Marx. Chain To study the probability of a single move requires that we know the number of offspring of each creature, so... Definitions$i$The number of offspring of an individual is random. $$P(\xi_{l}=k)=p_{k}$$ Natural one step from definition to probability $ \begin{aligned}p j}&=P(X_{n+1}=j\big|X_{n}=i\big)\&=P (1}22}cdots+\i}j\big)\end{aligned} $ The exact nature of the sub-resistence is natural. We're both starting with the most natural sense, starting with the mechanisms of change, and finally getting the probability matrix.

The probability distribution of the Marcov chain

Chapman-kolmogorov equation (C-K equation)

Set$X={X_{n},n\ge 0}$It's a horse chain on a state space S, and there is. $$p_{ij}^{(k+m)}(n)=\sum_{l\in S}p_{il}^{(k)}(n)p_{lj}^{(m)}(n+k),n,m,k\geq0,i,j\in S$$ Or in matrix form. $$\mathbf{P}^{(k+m)}(n)=\mathbf{P}^{(k)}(n)\mathbf{P}^{(m)}(n+k)$$ The conclusion of this equation tells us The Markov chain.$k$The probability of a step transfer is fully determined by the probability of a step transfer. If he was a Zilong Ma's chain, We just need to get it.$m=1$ We can move and move.$k$Step Transfer Export$k+1$Step shift If you take it again,$k=1$ You can export all kinds of transfer probability distributions. Intuitive interpretation System$n$from Status$i$Let's go, bye.$k+m$Step shift,$n+k+m$Time reached state$j$It's a good idea. System$n$from Status$i$Let's go. First.$k$Step shift,$n+k$When it reaches a certain point Intermediate state$l$Again.$n+k$from this midstate$l$Let's go.$m$Step shift,$n+k+m$Time reached state$j$, and the middle statel needs to be taken through the entire state space$S$ Note that the matrix indicates that it is possible to calculate multiple-step transfer probability matrices directly using the matrix product, and that it is better to do it in a series of times. Fine.

Initial distribution versus absolute distribution

Initial Distribution

The Maple Chain.$X={X_n}$. The status space is$S$ Remember $q_j^{(0)}=P(X_0=j)$, $j\in S$ Name of probability distribution ${q_j^{(0)}:j\in S}$ It's the initial distribution of the Maze Chain X. Vector $$\mathbf{q}^{(0)}=(q_1^{(0)},q_2^{(0)},\cdots q_j^{(0)},\cdots)$$ The initial distribution vector for the Marcov chain The initial distribution study is that we don't do any transfer, the initial time, the probability of each state.

Absolute distribution

The Maple Chain.$X={X_n}$ . The status space is$S$ Remember $q_{j}^{(n)}=\mathbf{P}(X_{n}=j)$I'm not sure. Name of probability distribution ${q_j^{(n)}:j\in S}$ It's the absolute distribution of the Maze Chain X. Vector $$\mathbf{q}^{(n)}=(q_1^{(n)},q_2^{(n)},\cdots q_j^{(n)},\cdots)$$ It's the absolute distribution of the Marcov chain. The absolute distribution study is the probability of a state of last resort at a time after constant transfer.

The link between the two

Zhiji Markov chain.$X$The absolute distribution is fully determined by its initial distribution and the probability of a step transfer. Probability form is $$q_j^{(n)}=\sum_{i\in S}q_i^{(0)}p_{ij}^{(n)}(0)$$ The matrix is as $${q}^{(n)}={q}^{(0)}{P}^{(n)}(0)$$ The Zilong Ma's chain can be removed. We use the CK equation to move from one step to another; this is the the theorem that can now export the absolute distribution of any time from the initial distribution.

Limited dimensions distribution

The Markov chain.$X$The limited dimensions of distribution are fully determined by their initial distribution and the probability of a step transfer We gave the formula there. $$P{X_{t_1}=i_1,X_{t_2}=i_2,\cdots,X_{t_n}=i_n}$$ $$=\sum_{i\in S}q_i^{(0)}\cdot p_{ii_1}^{(t_1)}(0)\cdot p_{ii_2}^{(t_2-t_1)}(t_1)\cdots p_{i_{n-1}i_n}^{(t_n-t_{n-1})}(t_{n-1}).$$ All the examples in this section are treated in the same way; whatever the problem, it turns into a demand.$k$The problem of moving probability matrices and initial distribution for absolute distribution; just figure out the problem.

Classification of the Marcov chain state

Status Type Definition

Study yourself. Definitions: $$f_{ij}^{(n)}=P{X_n=j,X_k\neq j,k=1,2,\cdots,n-1|X_0=i}$$ Yes$0$Time from Status$i$Let's go. $n$First time after step shift$j$The probability of a single-time probability. $$f_{ij}^{(+\infty)}=P{X_{n}\neq j,n=1,2,\cdots|X_{0}=i}$$ Yes$0$Time from Status$i$Let's go. Never get there.$j$Probability $$f_{ij}=\sum_{n=1}^{\infty}f_{ij}^{(n)}$$ Yes$0$Time from Status$i$We're moving out, and we're going to get to the state after a limited movement.$j$The probability of a time-out is called the probability of a time-out. Special $i=j$ Time $f_{ii}$ Yes$0$Time from Status$i$We're going to go back after a limited move.$i$Probability Definitions:

• If$f_{ii}=1$ And I'm a permanent return.
• If $f {ii}<$1 million, which is called state i is very retrograde, skimmed. If a state is always back, then there is.$\begin{aligned}f_{ii}=\sum_{n=1}^{\infty}f_{ii}^{(n)}=1\end{aligned}$ Which means it's a probability distribution. $$\mu_{ii}=\sum_{n=1}^{\infty}n\cdot f_{ii}^{(n)}$$ Average time called return If a state is always returned, if
• $\mu_{ii}<\infty$ says it's normal.
• $\mu_{ii}=\infty$ It's called a zero return. You can't just start with definitions.

The maximum number of conventions to be assembled by GCD functions $d i=mathrm{GCD}n\geq1, p ii}n}n}n}>I'm sorry If $d i>1$ 称为周期状态 周期为$d i$ If $d_i=1$ Called a non-cyclical state If Status$i$Normal return to non-cycle is called "performing " ; if normal return to cycle, call it " cyclical " Cycle is a concept of permanent return, very often not a study cycle

Status Type Adjudication

Theorem: For the Marx chain

• $i$It's always back.$(f_{ii}=1)$Other Organiser $\sum_{n=1}^{\infty}p_{ii}^{(n)}=+\infty$
• $i$Very good.<1)$充要条件为 $\sum_{n=1}^{\infty}p_{ii}^{(n)}<+infty This is the theory that allows us to...$n$The probability of a step shift begins to determine normality.

Theorem: Set-up status$i$It's always coming back, then.

• $i$It's a zero-sum condition.$\lim_{n\to\infty}p_{ii}^n=0$
• $i$It's a perfect condition for the waltz.>0$
• $i$It's a condition of normal return cycle.$\lim_{n\to\infty}p_{ii}^n$ Cannot initialise Evolution's mail component.

Inference: if$j$Yes, very or very often. Any situation is arbitrary.$i$ Yes. $$\lim_{n\to\infty}p_{ij}^{(n)}=0$$ Inference: for non-cyclical issues

• If it exists$n$ Let $p ii}n}n}>0,p_{ii}^{(n+1)}>0$ 则$i$/$/non-cycle
• If positive number exists$m$ Status$j$ Let $p ({\mmathrm}>0$则$j$ Non-cycle

Relationship between status

Studying two states together. Definitions: If it exists$n\ge 1$ Let $p j}n}>0$ 则称状态i可达状态j 记为 $I'm not a good guy. If two states are possible, it's called two states. Easy to verify

• It's transmissible.
• Interconnectivity is transmissible.
• Interconnectivity is symmetrical. Theoretically:
• $i\rightarrow j\Leftrightarrow f_{ij}>0$
• If I ever come back and... $i\rightarrow j$ There is. $f_{ji}=1$ So the two are connected. Theoretically: Two interoperable states: Either it's the same or it's the same or it's the same or it's the same or it's the same or it's the same. So we can extrapolate from one state to another.

Disaggregation of state space

It's easy to verify, to satisfy each other with self-reversible symmetry, transmission, that's an equal value relationship. Then we can divide the price classes. Because the state of interconnectivity is the same, and that's the basis for decomposition.

Theorem: contains equivalents of normal return Category$S_n$It's not closed. Set Theorem: state space of the chain$S$But the only way to break down is to divide into a limited or unlimited array of non-interconnected sub-sets, which is, $$S=D\cup C_1\cup C_2\cup\cdots $$ Of which$D$It's a very retrogressive subset of state. $C$It's all about the non-closure of a state of normality.

Theorem: X is a limited-state unicorn chain, and

• The X's very back-state set D can't be closed.
• X doesn't have a zero-return state.
• If X is not available, all X's state is returned. These are theorems that allow us to decompose the space, to decompose or to transfer the probability. Figure

Analysis of status from status map

The CK equation is too complicated. The state map is the easiest.

• All the people who have gone out of nowhere are very, very happy.
• It's always the same when you go out and you can go back to yourself.
• There's no one in the chain with a single one.
• The study cycle is being conducted by manual search for more than zero pii for maximum number of conventions
• Interconnective status is the same type of state

Maximum probability of diversion

The limit of the probability of a transfer is,$$\lim_{n\to\infty}P_{ij}^{(n)}$$including the existence and$i$Is it irrelevant?

Maximum distribution

Set$X={X_n,n=0,1,...}$For the second-martial chain, if any, $i,j$ $\lim_{n\to\infty}p_{ij}^{(n)}=\pi_j$, and \pi j>0,\sum_{j\in S}\pi_j=1$ $I'm sorry. then${\pi_j,j\in S}$It's a probability distribution, called the maximum distribution of the Mall chain. And the rest of the research is about this.

Distribution from state-of-the-art space research limit

When $iJ.A.A.T.A.T.A., right? $$\lim_{n\to\infty}p_{ij}^{(n)}=0$$ When $ij.$0.00 for normal return-cycle subsets Time $$\lim_{n\to\infty}p_{ij}^{(n)}\text{不存在}$$ Now let's start with the calculation. $$\lim_{n\to\infty}\color{}{p_{ij}^{(nd_j+r)}}$$ Give theorem:$j$It's normal to return, but... $$\lim_{n\to\infty}p_{ij}^{(nd_j+r)}=f_{ij}(r)\frac{d_j}{\mu_{jj}}$$ And to calculate this limit probability, you need to use it.$n$Stepover Period Average return time Inference: When the chain is an unattainable chain, for any dollar II've got it. $US$milm n\infory}p =(n)}\frac{1}{\mu}j}>$0.00 We just need to calculate the average return time, but we can keep it simple. Theorem: When the chain is an unattainable chain, for any dollar II've got it. $$\lim_{n\to\infty}p_{ij}^{(n)}=\frac{1}{\mu_{jj}}\stackrel{def}{=}\pi_{j}$$ And there is. $\pi_j$It's a linear equation group. $\begin{aligned}x_j=\sum_{i\in S}x_ip_{ij}\end{aligned}$ Conditions met$x_{j}\geq0,\sum_{j\in S}x_{j}=1$ The only solution. We'll calculate the maximum distribution later on, just the equation group, and then we'll use the penultimate to get back to the average time. The equation group, which we introduced in the algebra, can be solved slowly even with hand count.

A smooth distribution of transfer probability

Smooth distribution

Probability distribution$\pi_j$ It's a smooth distribution of the Zilong Ma's chain. $$\pi_j=\sum_{i\in S}\pi_ip_{ij},\quad j\in S$$ Or the matrix is. $$\pi=\pi P$$ of which$\pi$It's the limit distribution vector. $P$It's a transfer probability matrix. The smooth distribution is defined on the basis of the maximum distribution. Because a single shift is constant, there must be one. $$\pi_j=\sum_{i\in S}\pi_ip_{ij}^{(n)}$$ Theoretically: If$\pi$It's a smooth distribution of a unicorn chain.$\pi$For initial distribution yes One: $$P(X_n=i)=\sum_{k\in S}P(X_0=k)P(X_n=i\begin{vmatrix}X_0=k\end{vmatrix}$$ $$=\sum_{k\in S}\pi_{k}p_{ki}^{(n)}=\pi_{i}$$ Which means absolute distribution is constant. II: Making a decision$t,n,m,i$ Yes. $$P(X_{t_1+m}=i_1,\cdots,X_{t_n+m}=i_n)=P(X_{t_1}=i_1,\cdots,X_{t_n}=i_n)$$ The Matheric is a steady time series.

Studying the existence and calculation of smooth distribution

Unarranged Ma's chain - all the way through the chain.

Only limit distribution$${\pi_{j}=\frac{1}{\mu_{jj}},j\in S}$$The equations that calculate this flat distribution can be obtained from the first section's understanding equation.

Unaccepted Marx chain - cycle chain

Assumptions$X$It's a chain of maxs, and every state in the space is returned in a normal way.$d$There is.$X$There's only one smooth distribution.${\pi_{j}=\frac{1}{\mu_{ij}},j\in S}$ He can also solve the equation. Group $$00\ &\pi_{j}=\sum_{i\in S}\pi_{i}p_{ij} \ &== sync, corrected by elderman == @elder man I'm sorry, I'm sorry. Got it.

Usually a Zhiqikov chain.

Set Status Space to$S=D\cup C_{0}\cup C_{1}\cup\cdots$ of which$D$It's a very retrograde state. Set $C_0$It's a zero-sum situation. Set$C_m$It's normal to close the collection.$H=\bigcup_{k\geq1}C_{k}$ then

• $X$There is no requirement for a smooth distribution. $H=\Phi$
• $X$The only condition for a smooth distribution is a normal return to the closed-door. Set
• $X$There are numerous and smooth distributions, which require at least two normal return closures. Set Even if it's not the only one, we can still calculate the flat distribution, as follows: Example For seven states of Zithalkov, the one-step transfer probability matrix is $P=\begin{bmatrix}0.5&0.5&0&0&0&0\0&2/3&1/3&0&0&0&0\1/3&0&2/3&0&0&0&0\0&0&0&0.5&0.5&0&0\0&0&0&0.5&0.5&0&0\0&0&0&0&0&1&0\\frac{1}{7}&\frac{1}{7}&\frac{1}{7}&\frac{1}{7}&\frac{1}{7}&\frac{1}{7}&\frac{1}{7}\end{bmatrix}$$ 分解状态空间有 $$\begin{aligned}S&=D\cup C_1^+\cup C_2^+\cup C_3^+\\&={6}\cup{0,1,2}\cup{3,4}\cup{5}\end{aligned}$$ 所以有无穷多个平稳分布 使用分块矩阵 $$P_1=\begin{pmatrix}1/2&1/2&0\0&2/3&1\1/3&0&2/3\end{pmatrix}\quad P_2=\begin{pmatrix}1/2&1/2\1/2&1/2\end{pmatrix}\quad P_3=1$$ 解方程组有 $$\pi^{(1)}={\frac28,\frac38,\frac38}\quad\pi^{(2)}={\frac12,\frac12}\quad\pi^{(3)}={1}$$ 所以平稳分布为 $${\fnH00FFFF}\fradsymbol{\fnH00FF}\fscH00FF}\fscH00FF}\fscH00FFFF}\fscH00FF}\fscH00FF}\fscH00FF}\fscH00FF}\fscH00FFFF}\fscH00FF}\fsc(fscH00}\fsc(fscH00FF}\fsc(fsc)\fsc(fsc)\fsc(fsc)\fsc(fsc)\fsc(fscH00(fsc)\fsc(sc)\fsc(sc)\fsc(fsc(fsc)\sc(sc)\fsc(sc)\fsc(fsc(fsc)\fsc(fscsc)\fscsc(fscscsc)\f - There's one.$\lambda_1+\lambda_2+\lambda_3=1$ And it's easy to calculate on a smooth distribution basis, and we have a lot of good things to use.