人工神经网络

人工神经网络学习笔记，记录前馈传播信号和反向传播误差的矩阵表示

注意：神经网络输入层仅表示输入信号，输入节点不对输入值应用激活函数，历史就是这样规定的

前馈传播信号

人工神经网络层与层之间前馈传播方式：

$$X=W \cdot I$$

其中，W是权重矩阵，I是输入矩阵，X是组合调节后的信号，输入到下一层，然后应用激活函数，使神经网络的反应更像是生物神经元，激活函数为logistic function：$y=\frac{1}{1+e^{-x}}$

$$O=\operatorname{sigmoid}(X)$$

反向传播误差

忽略归一化因子分母，反向传播误差矩阵表示为：以输出层与挨着输出层的隐藏层为例，其他层之间也是一样的道理

$$\text { error }_{\text {hidden }}=W^{T}_{\text {hidden_output}} \cdot \text { error }_{\text {output }}$$

实践证明，这种相对简单的误差信号反馈方式，与我们先前相对复杂的方式一样有效！

Cost function：

如果神经网络有n个输出节点：

$$E = \sum_{n}\left(t_{n}-o_{n}\right)^{2}$$

对权重$w_{j, k}$的偏导数为

$$\frac{\partial E}{\partial w_{j, k}}=\frac{\partial}{\partial w_{j, k}}\sum_{n}\left(t_{n}-o_{n}\right)^{2}$$

因为神经网络的结构，节点k的输出不依赖于其他不与它相连的链接，这可以让我们删除令人厌烦的求和运算。这是一个很有用的技巧。代价函数根本就不需要对所有输出节点求和。原因是节点的输出只取决于所连接的链接。这个过程在许多教科书中一略而过，这些教科书只是简单地声明了误差函数，却没有解释原因。所以我们有：

$$\frac{\partial E}{\partial w_{j, k}}=\frac{\partial}{\partial w_{j, k}}\left(t_{k}-o_{k}\right)^{2}$$

gradient descent：

这里我们以三层，每层三个节点的神经网络为例，用i表示输入层，j表示隐藏层，k表示输出层，利用链式法则：
$\frac{\partial E}{\partial w{j, k}}=\frac{\partial E}{\partial o{k}} \cdot \frac{\partial o{k}}{\partial w{j, k}}$
$\frac{\partial E}{\partial w{j, k}}=-2\left(t{k}-o{k}\right) \cdot \frac{\partial o{k}}{\partial w{j, k}}$
$\frac{\partial E}{\partial w{j, k}}=-2\left(t{k}-o{k}\right) \cdot \frac{\partial}{\partial w{j, k}} \operatorname{sigmoid}\left(\Sigma{j} w{j, k} \cdot o{j}\right)$
$\frac{\partial E}{\partial w{j, k}}=-2\left(t{k}-o{k}\right) \cdot \operatorname{sigmoid}\left(\Sigma{j} w{j, k} \cdot o{j}\right)\left(1-\operatorname{sigmoid}\left(\Sigma{j} w{j, k} \cdot o{j}\right)\right) \cdot \frac{\partial}{\partial w{j, k}}\left(\Sigma{j} w{j, k} \cdot o{j}\right)$
$=-2\left(t{k}-o{k}\right) \cdot \operatorname{sigmoid}\left(\Sigma{j} w{j, k} \cdot o{j}\right)\left(1-\operatorname{sigmoid}\left(\Sigma{j} w{j, k} \cdot o{j}\right)\right) \cdot o{j}$

同理，对于输入层和隐藏层，我们有

$$\frac{\partial E}{\partial w_{i j}}=-\left(e_{j}\right) \cdot \operatorname{sigmoid}\left(\Sigma_{i} w_{i j} \cdot o_{i}\right)\left(1-\operatorname{sigmoid}\left(\Sigma_{i} w_{i j} \cdot o_{i}\right)\right) \cdot o_{i}$$

梯度的矩阵形式：

$$\left(\begin{array}{cccc} \Delta w_{1,1} & \Delta w_{2,1} & \Delta w_{3,1} & \cdots \\ \Delta w_{1,2} & \Delta w_{2,2} & \Delta w_{3,2} & \cdots \\ \Delta w_{1,3} & \Delta w_{2,3} & \Delta w_{1 j} k & \cdots \\ \cdots & \cdots & \cdots & \cdots \end{array}\right) = \left(\begin{array}{cccc} E_{1} * S_{1}\left(1-S_{1}\right) \\ E_{2} * S_{2}\left(1-S_{3}\right) \\ \cdots \\ E_{k} * S_{k}\left(1-S_{k}\right) \\ \cdots \end{array}\right) \cdot \left(\begin{array}{cccc} O_{1} & O_{2} & \dots & O_{j} & \dots \end{array}\right)$$

梯度的反向传播

cost function对输出层输入值$z{3}$的梯度$\nabla{z^{(3)}} J_{1}$

我们以手写数字识别为例，手写字体为$20 \times 20$像素的灰度值图片，神经网络结构为三层，第一层为输入层，为401维$(20 \times 20+1)$，第二层为隐藏层，25+1维，第三层是输出层，10维，每一个输出代表第k类的概率。神经网络前馈传播信号过程如下：

$$\stackrel{a^{(1)} = x}{\longrightarrow} \left(\begin{array}{cccc} 1 \\ x_{1} \\ x_{2} \\ \dots \\ x_{400} \end{array}\right) \stackrel{z^{(2)} = \Theta^{(1)}a^{(1)}}{\longrightarrow} \left(\begin{array}{cccc} z_{1}^{(2)} \\ z_{2}^{(2)} \\ \dots \\ z_{25}^{(2)} \end{array}\right) \stackrel{a^{(2)} = g \left( z^{(2)} \right)}{\longrightarrow} \left(\begin{array}{cccc} 1 \\ a^{(2)}_{1} \\ a^{(2)}_{2} \\ \dots \\ a^{(2)}_{25} \end{array}\right) \stackrel{z^{(3)} = \Theta^{(2)}a^{(2)}}{\longrightarrow} \left(\begin{array}{cccc} z_{1}^{(3)} \\ z_{2}^{(3)} \\ \dots \\ z_{10}^{(3)} \end{array}\right) \stackrel{a^{(3)} = g\left(z^{(3)}\right)=h_{\theta}(x)}{\longrightarrow} \left(\begin{array}{cccc} h_{\theta}(x)_{1} \\ h_{\theta}(x)_{2} \\ \dots \\ h_{\theta}(x)_{10} \end{array}\right)$$

神经网络的cost function:

$$\begin{aligned} J(\theta)=& \frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K}\left[-y_{k}^{(i)} \log \left(\left(h_{\theta}\left(x^{(i)}\right)\right)_{k}\right)-\left(1-y_{k}^{(i)}\right) \log \left(1-\left(h_{\theta}\left(x^{(i)}\right)\right)_{k}\right)\right]+ \\ &\frac{\lambda}{2 m}\left[\sum_{j=1}^{25} \sum_{k=1}^{400}\left(\Theta_{j, k}^{(1)}\right)^{2}+\sum_{j=1}^{10} \sum_{k=1}^{25}\left(\Theta_{j, k}^{(2)}\right)^{2}\right] \end{aligned}$$

为了方便，我们把cost function的前半部分用$J_{1}$表示:

$$J_{1}=\frac{1}{m} \sum_{k=1}^{K}\left[-y_{k}^{(i)} \log \left(\left(h_{\theta}\left(x^{(i)}\right)\right)_{k}\right)-\left(1-y_{k}^{(i)}\right) \log \left(1-\left(h_{\theta}\left(x^{(i)}\right)\right)_{k}\right)\right]$$

因为
$\frac{\partial ln(h{\theta}(x){k})}{\partial z^{(3)}{k}}=\frac{1}{h{\theta}(x){k}}\frac{dh{\theta}(x){k}}{dz^{(3)}{k}}=\frac{1}{h{\theta}(x){k}}h{\theta}(x){k}(1-h{\theta}(x){k})=1-h{\theta}(x){k}$
$\frac{\partial ln(1-h{\theta}(x){k})}{\partial z^{(3)}{k}}=\frac{-1}{1-h{\theta}(x){k}}\frac{dh{\theta}(x){k}}{dz^{(3)}{k}}=\frac{-1}{1-h{\theta}(x){k}}h{\theta}(x){k}(1-h{\theta}(x){k})=-h{\theta}(x){k}$
$J{1}$虽然有求和符号，但是求梯度分量的过程中我们知道，第k个分量$z^{(3)}{k}$只与$h_{k}$有关，而与其他分量无关。所以求和符号在求梯度过程中可省略掉

$$\frac{\partial J_{1}}{\partial z^{(3)}_{k}} = -y_{k} (1-h_{\theta}(x)_{k}) - (1-y_{k})(-h_{\theta}(x)_{k}) = h_{\theta}(x)_{k}-y_{k}$$

$J_{1}$对$z^{(3)}$求梯度，经过化简最终我们有：

$$\nabla_{z^{(3)}} J_{1} = \vec{h_{\theta}} - \vec{y}$$

我们成功地求出了对输出层z3的梯度！而且对数似然链式求导和logistic函数链式求导后会约掉，数学之美！简洁之美！

cost function对隐藏层输出值$a{2}$的梯度$\nabla{a^{(2)}} J_{1}$

在前馈传播过程中，我们有：

$$z^{(3)} = \Theta^{(2)}a^{(2)}$$

接下来结论很重要，梯度的反向传播的求法：

我们以2维向量前馈传播为例，高维向量一样的道理！

$$\left(\begin{array}{c}y_{1} \\ y_{2} \end{array}\right) = \left(\begin{array}{c}a&b \\ c&d \end{array}\right) \left(\begin{array}{c}x_{1} \\ x_{2} \end{array}\right)=\Theta \left(\begin{array}{c}x_{1} \\ x_{2} \end{array}\right)$$

其中，$\vec{y}$是神经网络在下一层输入，$\vec{x}$是前一层输出，$\Theta$是权重矩阵

$$\Theta = \left(\begin{array}{c}a&b \\ c&d \end{array}\right)$$

我们有

$$\left\{ \begin{aligned} y_{1} = a x_{1} + bx_{2} \\ y_{2} = c x_{1} + dx_{2} \end{aligned} \right.$$

cost function对向量$\vec{x}$求梯度，有

$$\nabla_{x} J_{1} = \left(\begin{array}{c}\frac{\partial J_{1}}{\partial x_{1}} \\ \frac{\partial J_{1}}{\partial x_{2}} \end{array}\right) = \left(\begin{array}{c}\frac{\partial J_{1}}{\partial y_{1}}a + \frac{\partial J_{1}}{\partial y_{2}}c \\ \frac{\partial J_{1}}{\partial y_{1}}b+\frac{\partial J_{1}}{\partial y_{2}}d \end{array}\right) = \left(\begin{array}{c}a&c\\ b&d \end{array}\right)\left(\begin{array}{c}\frac{\partial J_{1}}{\partial y_{1}} \\ \frac{\partial J_{1}}{\partial y_{2}} \end{array}\right) = \Theta^{T}\left(\begin{array}{c}\frac{\partial J_{1}}{\partial y_{1}} \\ \frac{\partial J_{1}}{\partial y_{2}} \end{array}\right)$$

所以有

$$\nabla_{a_{2}} J_{1} = \Theta^{T} \nabla_{z_{3}} J_{1}$$

梯度$\nabla{z^{(3)}} J{1}$ 与参数$\Theta{2}$的梯度$\nabla{\Theta{2}} J{1}$的关系：

在前馈传播过程中，我们有：

$$z^{(3)} = \Theta^{(2)}a^{(2)}$$

我们还是以2维向量前馈传播为例，高维向量一样的道理！

$$\left(\begin{array}{c}y_{1} \\ y_{2} \end{array}\right) = \left(\begin{array}{c}a&b \\ c&d \end{array}\right) \left(\begin{array}{c}x_{1} \\ x_{2} \end{array}\right)=\Theta \left(\begin{array}{c}x_{1} \\ x_{2} \end{array}\right)$$

其中，$\vec{y}$是神经网络在下一层输入，$\vec{x}$是前一层输出，$\Theta$是权重矩阵

$$\Theta = \left(\begin{array}{c}a&b \\ c&d \end{array}\right)$$

我们有

$$\left\{ \begin{aligned} y_{1} = a x_{1} + bx_{2} \\ y_{2} = c x_{1} + dx_{2} \end{aligned} \right.$$

cost function对$\Theta_{2}$求梯度，有

$$\nabla_{\Theta_{2}} J_{1} = \left(\begin{array}{c}\frac{\partial J_{1}}{\partial a} &\frac{\partial J_{1}}{\partial b} \\ \frac{\partial J_{1}}{\partial c}&\frac{\partial J_{1}}{\partial d} \end{array}\right) = \left(\begin{array}{c}\frac{\partial J_{1}}{\partial y_{1}}x_{1}&\frac{\partial J_{1}}{\partial y_{1}}x_{2} \\ \frac{\partial J_{1}}{\partial y_{2}}x_{1}&\frac{\partial J_{1}}{\partial y_{2}}x_{2} \end{array}\right) = \left(\begin{array}{c}\frac{\partial J_{1}}{\partial y_{1}} \\ \frac{\partial J_{1}}{\partial y_{2}} \end{array}\right) \left(\begin{array}{c} x_{1}&x_{2} \end{array}\right) = \nabla_{y} J_{1} \left(\begin{array}{c} x_{1}&x_{2} \end{array}\right)$$

我们成功地求出了对参数$\Theta_{2}$的梯度！

$$\nabla_{\Theta_{2}} J_{1} = \nabla_{z_{3}} J_{1} \cdot a_{2}^{T}$$

cost function对隐藏层输入值$z{2}$的梯度$\nabla{z^{(2)}} J_{1}$

在前馈传播过程中，我们有：

$$a_{(2)} = sigmoid(z_{(2)})$$

对分量求偏导

$$\frac{\partial J_{1}}{\partial z^{(2)}_{k}}=\frac{\partial J_{1}}{\partial a^{(2)}_{k}} \frac{\partial a^{(2)}_{k}}{\partial z^{(2)}_{k}} =\frac{\partial J_{1}}{\partial a^{(2)}_{k}} g(z^{(2)}_{k})(1-g(z^{(2)}_{k}))$$

把分量整合成向量的形式

$$\nabla_{z_{2}} J_{1} = \nabla_{a_{2}} J_{1} \ast g(z^{(2)})(1-g(z^{(2)}))$$

cost function对参数$\Theta{1}$的梯度$\nabla{\Theta{(1)}} J{1}$

在前馈传播过程中，我们有：

$$z_{(2)} = \Theta^{(1)} a^{(1)}$$ $$\nabla_{\Theta_{1}} J_{1} = \nabla_{z_{z}} J_{1} \cdot a_{1}^{T}$$