多项分布

$p(\mathbf{x} \mid \boldsymbol{\mu})=\prod_{k=1}^K \mu_k^{x_k}$

• $x_k$ 中一项为 1，其余为 0

期望

• $\mathbb{E}[\mathbf{x} \mid \boldsymbol{\mu}]=\sum_{\mathbf{x}} p(\mathbf{x} \mid \boldsymbol{\mu}) \mathbf{x}=\left(\mu_1, \ldots, \mu_M\right)^{\mathrm{T}}=\boldsymbol{\mu}$

似然函数

• $p(\mathcal{D} \mid \boldsymbol{\mu})=\prod_{n=1}^N \prod_{k=1}^K \mu_k^{x_{n k}}=\prod_{k=1}^K \mu_k^{\left(\sum_n x_{n k}\right)}=\prod_{k=1}^K \mu_k^{m_k}$

  • $m_k=\sum_n x_{n k}$

极大似然估计

• 对 p 求对数似然，约束项是 $\sum_{\mathbf{x}} p(\mathbf{x} \mid \boldsymbol{\mu})=\sum_{k=1}^K \mu_k=1$ ，再通过 [[Lagrange Multiplier]] 求解

  • $L=\sum_{k=1}^K m_k \ln \mu_k+\lambda\left(\sum_{k=1}^K \mu_k-1\right)$

  • 对 L 求关于 $\mu_k$ 的导数，并令其为 0 。得到 $\mu_k=\frac{m_k}{N}$

    • $\begin{aligned} \frac{\partial L}{\partial \mu_k} & =\frac{m_k}{\mu_k}+\lambda \\ 0 & =\frac{m_k}{\mu_k}+\lambda \\ -\frac{m_k}{\lambda} & =\mu_k \\ -\frac{1}{\lambda} \sum_k m_k & =\sum_k \mu_k \\ -\frac{N}{\lambda} & =1 \\ \lambda & =-N\end{aligned}$

重复进行 N 次多项分布实验，得到分布

• $\operatorname{Mult}\left(m_1, m_2, \ldots, m_K \mid \boldsymbol{\mu}, N\right)=\left(\begin{array}{c}N \\ m_1 m_2 \ldots m_K\end{array}\right) \prod_{k=1}^K \mu_k^{m_k}$

  • $\left(\begin{array}{c}N \\ m_1 m_2 \ldots m_K\end{array}\right)=\frac{N !}{m_{1} ! m_{2} ! \ldots m_{K} !}$.

  • $\sum_{k=1}^K m_k=N$