含义是原装的，不是变体，可以理解为原装T-34，不是后来魔改的版本；
下面以
gradiant descent为例来翻译翻译什么tmd叫tmd
vanilla；
Vanilla Gradient Descent
模型定义
y ^ = X W + b \begin{equation} \hat{y} = \mathbf{X} \mathbf{W} + b \end{equation} y^=XW+b
损失函数
J ( W , b ) = 1 m ∑ i = 1 m ( y ( i ) − y ^ ( i ) ) 2 \begin{equation} J(\mathbf{W}, b) = \frac{1}{m} \sum_{i=1}^{m} (y^{(i)} - \hat{y}^{(i)})^2 \end{equation} J(W,b)=m1i=1∑m(y(i)−y^(i))2
梯度计算
∂ J ( W , b ) ∂ W = − 2 m X T ( y − y ^ ) \begin{equation} \frac{\partial J(\mathbf{W}, b)}{\partial \mathbf{W}} = -\frac{2}{m} \mathbf{X}^T (\mathbf{y} - \hat{y}) \end{equation} ∂W∂J(W,b)=−m2XT(y−y^)
∂ J ( W , b ) ∂ b = − 2 m ∑ i = 1 m ( y ( i ) − y ^ ( i ) ) \begin{equation} \frac{\partial J(\mathbf{W}, b)}{\partial b} = -\frac{2}{m} \sum_{i=1}^{m} (y^{(i)} - \hat{y}^{(i)}) \end{equation} ∂b∂J(W,b)=−m2i=1∑m(y(i)−y^(i))
参数更新
W : = W − α ∂ J ( W , b ) ∂ W \begin{equation} \mathbf{W} := \mathbf{W} - \alpha \frac{\partial J(\mathbf{W}, b)}{\partial \mathbf{W}} \end{equation} W:=W−α∂W∂J(W,b)
b : = b − α ∂ J ( W , b ) ∂ b \begin{equation} b := b - \alpha \frac{\partial J(\mathbf{W}, b)}{\partial b} \end{equation} b:=b−α∂b∂J(W,b)
Momentum Gradient Descent
模型定义
y ^ = X W + b \begin{equation} \hat{y} = \mathbf{X} \mathbf{W} + b \end{equation} y^=XW+b
损失函数
J ( W , b ) = 1 m ∑ i = 1 m ( y ( i ) − y ^ ( i ) ) 2 \begin{equation} J(\mathbf{W}, b) = \frac{1}{m} \sum_{i=1}^{m} (y^{(i)} - \hat{y}^{(i)})^2 \end{equation} J(W,b)=m1i=1∑m(y(i)−y^(i))2
梯度计算
∂ J ( W , b ) ∂ W = − 2 m X T ( y − y ^ ) \begin{equation} \frac{\partial J(\mathbf{W}, b)}{\partial \mathbf{W}} = -\frac{2}{m} \mathbf{X}^T (\mathbf{y} - \hat{y}) \end{equation} ∂W∂J(W,b)=−m2XT(y−y^)
∂ J ( W , b ) ∂ b = − 2 m ∑ i = 1 m ( y ( i ) − y ^ ( i ) ) \begin{equation} \frac{\partial J(\mathbf{W}, b)}{\partial b} = -\frac{2}{m} \sum_{i=1}^{m} (y^{(i)} - \hat{y}^{(i)}) \end{equation} ∂b∂J(W,b)=−m2i=1∑m(y(i)−y^(i))
参数更新
v W = β v W + ( 1 − β ) ∂ J ( W , b ) ∂ W \begin{equation} v_{\mathbf{W}} = \beta v_{\mathbf{W}} + (1 - \beta) \frac{\partial J(\mathbf{W}, b)}{\partial \mathbf{W}} \end{equation} vW=βvW+(1−β)∂W∂J(W,b)
v b = β v b + ( 1 − β ) ∂ J ( W , b ) ∂ b \begin{equation} v_{b} = \beta v_{b} + (1 - \beta) \frac{\partial J(\mathbf{W}, b)}{\partial b} \end{equation} vb=βvb+(1−β)∂b∂J(W,b)
W : = W − α v W \begin{equation} \mathbf{W} := \mathbf{W} - \alpha v_{\mathbf{W}} \end{equation} W:=W−αvW
b : = b − α v b \begin{equation} b := b - \alpha v_{b} \end{equation} b:=b−αvb
总之，vanilla表示是初始的样子，可以理解为baseline，后面一堆魔改的方法喜欢和它做比较；