指数分布族

Define1

称一个概率密度函数族或概率质量函数族为指数族（exponential family)，如果它能统一表示为:

f (x ∣ θ) = h (x) c (θ) \exp (\sum_{i = 1}^{k} w_{i} (θ) t_{i} (x))

$θ$ 是向量值参数

Example 1 （二项指数族）

二项分布族满足 X~b（n，p），又称 Bernoulli 分布
该分布族的概率密度函数为：

\begin{aligned} f (x ∣ p) = & (\binom{n}{x}) p^{x} (1 - p)^{n - x} \\ = & (\binom{n}{x}) (1 - p)^{n} {(\frac{p}{1 - p})}^{x} \\ = & (\binom{n}{x}) (1 - p)^{n} \exp (\log (\frac{p}{1 - p}) x) \end{aligned}

显然，$$\begin{aligned}&h(x)=\begin{cases}\binom nx,&x=0,\cdots,n\0,&\text{其他}\end{cases}，{c(p)=(1-p)^n,0<p<1}\&w_1(p)=\log\left(\frac p{1-p}\right),0<p<1,\quad\text{以及 }t_1(x)=x\end{aligned}$$

Example 2

Theorem 1 (指数族的矩)

$\begin{matrix} E (\sum_{i = 1}^{k} \frac{\partial w_{i} (θ)}{\partial θ_{j}} t_{i} (X)) = - \frac{\partial}{\partial θ_{j}} \log c (θ) \\ Var (\sum_{i = 1}^{k} \frac{\partial w_{i} (θ)}{\partial θ_{j}} t_{i} (X)) = - \frac{\partial^{2}}{\partial θ_{j}^{2}} \log c (θ) - E (\sum_{i = 1}^{k} \frac{\partial^{2} w_{i} (θ)}{\partial θ_{j}^{2}} t_{i} (X)) \end{matrix}$

Proof 1

E (\sum_{i = 1}^{k} \frac{\partial w_{i} (θ)}{\partial θ_{j}} t_{i} (X)) = - \frac{\partial}{\partial θ_{j}} \log c (θ)

定义似然函数

L (x, θ) = \log f (x ∣ θ) = \log h (x) + \log c (θ) + (\sum_{i = 1}^{k} w_{i} (θ) t_{i} (x))

令

\begin{aligned} U_{j} (x, θ) & = \frac{\partial}{\partial θ_{j}} L (x, θ) \\ = \frac{\partial}{\partial θ_{j}} \log c (θ) + (\sum_{i = 1}^{k} \frac{\partial w_{i} (θ)}{\partial θ_{j}} t_{i} (X)) \end{aligned}

只要证明 $E (U_{j}) = 0$ ，原题目得证。

Proof 1.1 (

E (U_{j}) = 0

)

\begin{aligned} E (U_{j}) & = \int_{- \infty}^{+ \infty} \frac{\partial}{\partial θ_{j}} \log f (x ∣ θ) f (x ∣ θ) d x \\ = \int_{- \infty}^{+ \infty} \frac{\partial}{\partial θ_{j}} f (x ∣ θ) d x \\ = \frac{\partial}{\partial θ_{j}} \int_{- \infty}^{+ \infty} f (x ∣ θ) d x \\ = 0 \end{aligned}

Proof 2

Var (\sum_{i = 1}^{k} \frac{\partial w_{i} (θ)}{\partial θ_{j}} t_{i} (X)) = - \frac{\partial^{2}}{\partial θ_{j}^{2}} \log c (θ) - E (\sum_{i = 1}^{k} \frac{\partial^{2} w_{i} (θ)}{\partial θ_{j}^{2}} t_{i} (X))

\begin{aligned} E (U_{j}^{2}) & = \int_{- \infty}^{+ \infty} {[\frac{\partial}{\partial θ_{j}} \log c (θ) + (\sum_{i = 1}^{k} \frac{\partial w_{i} (θ)}{\partial θ_{j}} t_{i} (X))]}^{2} f (x ∣ θ) d x \\ = Var (\sum_{i = 1}^{k} \frac{\partial w_{i} (θ)}{\partial θ_{j}} t_{i} (X)) \end{aligned}

注意到等式右边

- \frac{\partial^{2}}{\partial θ_{j}^{2}} \log c (θ) - E (\sum_{i = 1}^{k} \frac{\partial^{2} w_{i} (θ)}{\partial θ_{j}^{2}} t_{i} (X)) = E (- \frac{\partial}{\partial θ_{j}} U_{j} (x, θ))

故原问题化为：证

E (U^{2}) = E (- \frac{\partial}{\partial θ_{j}} U_{j} (x, θ))

\begin{aligned} E (- \frac{\partial}{\partial θ_{j}} U_{j}) & = \int_{- \infty}^{+ \infty} - \frac{\partial^{2}}{\partial θ_{j}^{2}} \log f f d x \\ = \int_{- \infty}^{+ \infty} - \frac{\partial}{\partial θ_{j}} (\frac{1}{f} f_{θ_{j}}) f d x \\ = \int_{- \infty}^{+ \infty} (\frac{1}{f^{2}} f_{θ_{j}} - \frac{1}{f} \frac{\partial}{\partial θ_{j}} f_{θ_{j}}) f d x \\ = E (U^{2}) - \int_{- \infty}^{+ \infty} \frac{\partial^{2}}{\partial θ_{j}^{2}} f d x \\ = E (U^{2}) \end{aligned}

Q.E.D