【白板推导系列笔记】线性分类-高斯判别分析（Gaussian Discriminant Analysis）-模型定义

{ ( x i , y i ) } i = 1 N , x i ∈ R p , y i ∈ { 0 , 1 }

{(xi​,yi​)}i=1N​,xi​∈Rp,yi​∈{0,1}​
逻辑回归是直接对$p(y|x)$建模，而高斯判别分析作为概率生成模型，是通过引入类型的先验，通过贝叶斯公式，得到联合分布$p(x,y)=p(x|y)p(y)$，再对联合分布的对数似然得到参数

贝叶斯公式为
$$p(y|x)=\frac{p(x|y)p(y)}{p(x)}$$

但是由于我们只关心 p ( y = 1 ∣ x ) = p ( x ∣ y = 1 ) p ( y = 1 ) p ( x )

p(y=1∣x)=p(x)p(x∣y=1)p(y=1)​​和 p ( y = 0 ∣ x ) = p ( x ∣ y = 0 ) p ( y = 0 ) p ( x )

p(y=0|x)=p(x|y=0)p(y=0)p(x)" role="presentation">p(y=0|x)=p(x|y=0)p(y=0)p(x)$$\begin{array}{r}p(y=0|x)=\frac{p(x|y=0)p(y=0)}{p(x)}\end{array}$$

p(y=0∣x)=p(x)p(x∣y=0)p(y=0)​​的大小关系，因此不需要关注分母，因为二者是一样的，即
y ^ = a r g m a x   y ∈ { 0 , 1 } p ( y ∣ x ) 由于 p ( y ∣ x ) ∝ p ( x ∣ y ) p ( y ) = a r g m a x   y p ( y ) ⋅ p ( x ∣ y )

y^=argmax y∈{0,1}⁡p(y|x)由于p(y|x)∝p(x|y)p(y)=argmax y⁡p(y)⋅p(x|y)" role="presentation">y^=argmax y∈{0,1}p(y|x)由于p(y|x)∝p(x|y)p(y)=argmax yp(y)⋅p(x|y)$$\begin{array}{rl}\hat{y}& =\underset{y\in \left\{0,1\right\}}{argmax}p(y|x)\\ & 由于p(y|x)\propto p(x|y)p(y)\\ & =\underset{y}{argmax}p(y)\cdot p(x|y)\end{array}$$

y^​​=y∈{0,1}argmax ​p(y∣x)由于p(y∣x)∝p(x∣y)p(y)=yargmax ​p(y)⋅p(x∣y)​
高斯判别分析我们对数据集作出的假设有，类的先验是二项分布，每一类的似然是高斯分布，即
y ∼ B ( 1 , ϕ ) ⇒ p ( y ) = { ϕ y y = 1 ( 1 − ϕ ) 1 − y y = 0 ⇒ p ( y ) = ϕ y ( 1 − ϕ ) 1 − y x ∣ y = 1 ∼ N ( μ 1 , Σ ) x ∣ y = 0 ∼ N ( μ 2 , Σ ) ⇒ p ( x ∣ y ) = N ( μ 1 , Σ ) y ⋅ N ( μ 2 , Σ ) 1 − y

\begin{aligned} y & \sim B(1,\phi)\Rightarrow p(y)=\left\{\begin{aligned}&\phi^{y}&y=1\\&(1-\phi)^{1-y}&y=0\end{aligned}" role="presentation">\begin{aligned} y & \sim B(1,\phi)\Rightarrow p(y)=\left\{\begin{aligned}&\phi^{y}&y=1\\&(1-\phi)^{1-y}&y=0\end{aligned}$$\text{begin{aligned} y \& sim B(1,phi)Rightarrow p(y)=left{begin{aligned}\&phi^y\&y=1\&(1-phi)^{1-y}\&y=0end{aligned}}$$

\right.\\ &\Rightarrow p(y)=\phi^{y}(1-\phi)^{1-y}\\ x|y=1 &\sim N(\mu_{1},\Sigma)\\ x|y=0 & \sim N(\mu_{2},\Sigma) \\ &\Rightarrow p(x|y)=N(\mu_{1},\Sigma)^{y}\cdot N(\mu_{2},\Sigma)^{1-y} \end{aligned} yx∣y=1x∣y=0​∼B(1,ϕ)⇒p(y)={​ϕy(1−ϕ)1−y​y=1y=0​⇒p(y)=ϕy(1−ϕ)1−y∼N(μ1​,Σ)∼N(μ2​,Σ)⇒p(x∣y)=N(μ1​,Σ)y⋅N(μ2​,Σ)1−y​
因此，最大后验
L ( μ 1 , μ 2 , Σ , ϕ ) = log ⁡ ∏ i = 1 N [ p ( x i ∣ y i ) p ( y i ) ] = ∑ i = 1 N [ log ⁡ p ( x i ∣ y i ) + log ⁡ p ( y i ) ] = ∑ i = 1 N [ log ⁡ N ( μ 1 , Σ ) y i + log ⁡ N ( μ 2 , Σ ) 1 − y i + log ⁡ ϕ y i ( 1 − ϕ ) 1 − y i ]

L(μ1,μ2,Σ,ϕ)=log⁡∏i=1N[p(xi|yi)p(yi)]=∑i=1N[log⁡p(xi|yi)+log⁡p(yi)]=∑i=1N[log⁡N(μ1,Σ)yi+log⁡N(μ2,Σ)1−yi+log⁡ϕyi(1−ϕ)1−yi]" role="presentation">L(μ1,μ2,Σ,ϕ)=log∏i=1N[p(xi|yi)p(yi)]=∑i=1N[logp(xi|yi)+logp(yi)]=∑i=1N[logN(μ1,Σ)yi+logN(μ2,Σ)1−yi+logϕyi(1−ϕ)1−yi]$$\begin{array}{rl}L({\mu }_1,{\mu }_2,\mathrm{\Sigma },\varphi )& =\log \prod _{i=1}^N[p(x_i|y_i)p(y_i)]\\ & =\sum _{i=1}^N[\log p(x_i|y_i)+\log p(y_i)]\\ & =\sum _{i=1}^N[\log N({\mu }_1,\mathrm{\Sigma }{)}^{y_i}+\log N({\mu }_2,\mathrm{\Sigma }{)}^{1-y_i}+\log {\varphi }^{y_i}(1-\varphi {)}^{1-y_i}]\end{array}$$

L(μ1​,μ2​,Σ,ϕ)​=logi=1∏N​[p(xi​∣yi​)p(yi​)]=i=1∑N​[logp(xi​∣yi​)+logp(yi​)]=i=1∑N​[logN(μ1​,Σ)yi​+logN(μ2​,Σ)1−yi​+logϕyi​(1−ϕ)1−yi​]​

CSDN话题挑战赛第2期
参赛话题：学习笔记