激活函数 Relu,Gelu,Mish,SiLU,Swish,Tanh,Sigmoid

我是胡歌

13832人浏览 · 2022-11-02 11:17:20

我是胡歌 · 2022-11-02 11:17:20 发布

Relu (Rectified Linear Unit)
$R e l u (x) = ma x (0, x)$

from torch import nn
import torch
import matplotlib
matplotlib.use('agg')
import matplotlib.pyplot as plt

func = nn.ReLU()
x = torch.arange(start=-2, end=2, step=0.01)
y = func(x)
plt.plot(x.numpy(), y.numpy())
plt.title("relu")
plt.savefig("relu.png")

在这里插入图片描述

2.Sigmoid
$Sigmoid(x)=\frac{1}{1+e^{−x}}$

func = nn.Sigmoid()
x = torch.arange(start=-10, end=10, step=0.01)
y = func(x)
plt.plot(x.numpy(), y.numpy())
plt.title("sigmoid")
plt.savefig("sigmoid.png")

在这里插入图片描述

3.Tanh
$sinh(x)=\frac{e^x−e^{−x}}{2}$
$cosh(x)=\frac{e^x+e^{−x}}{2}$
$t anh (x) = s inh (x) cos h (x)$

func = nn.Tanh()
x = torch.arange(start=-10, end=10, step=0.01)
y = func(x)
plt.plot(x.numpy(), y.numpy())
plt.title("tanh")
plt.savefig("tanh.png")

在这里插入图片描述

4.Silu(Sigmoid Linear Unit) or Swish

The SiLU activation function was introduced in "Gaussian Error Linear Units (GELUs)"Hendrycks et al. 2016and "Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning"Elfwing et al. 2017and was independently discovered (and called swish) in "Searching for Activation Functions"Ramachandran et al. 2017

$silu(x)=x∗sigmoid(x)=\frac{x}{1+e^{−x}}$

func = nn.Sigmoid()
x = torch.arange(start=-10, end=10, step=0.01)
y = func(x) * x
plt.plot(x.numpy(), y.numpy())
plt.title("silu")
plt.savefig("silu.png")

在这里插入图片描述

5.Gelu(Gaussian Error Linear Units)

$Gelu(x)=xP(X≤x)=xΦ(x)=x⋅\frac{1}{2}[1+erf(\frac{x}{\sqrt{2}})]$
其中， Φ(x) 是高斯分布的累计分布函数；其中误差函数
$erf(x)=\frac{2}{\sqrt{π}} \int_{0}^{z} e^{−t^2}dt$
gelu在精度要求不高的情况下，可用下列函数估计
$Gelu(x)≈0.5x(1+tanh[\sqrt{\frac{2}{\pi}}(x+0.044715x^3)])$
或者
$G e l u (x) \approx x \cdot s i g m o i d (1.702 x)$

func = torch.nn.functional.gelu
x = torch.arange(start=-10, end=10, step=0.01)
y = func(x) 
plt.plot(x.numpy(), y.numpy())
plt.title("gelu")
plt.savefig("gelu.png")

在这里插入图片描述

6.Mish (A Self Regularized Non-Monotonic Activation Function)

$\cdot tanh(softplus(x))=x \cdot tanh(ln(1+e^x))$

def mish(x):
    return x * torch.tanh(torch.nn.functional.softplus(x))
x = torch.arange(start=-10, end=10, step=0.01)
y = mish(x) 
plt.plot(x.numpy(), y.numpy())
plt.title("mish")
plt.savefig("mish.png")

在这里插入图片描述