深度学习发展基础

1. 大数据
2. 激活函数
3. DropOut
4. BatchNorm
5. ResNet
6. 初始化
7. 深度学习框架 caffe/Pytorch/tensorflow
8. 计算能力 （显卡）
9. …

全连接层

x = tf.random.normal([4, 784])
net = tf.keras.layers.Dense(512)
out = net(x)

out.shape   # (4, 512)
net.kernel.shape, net.bias.shape   # (784, 512) (512,)

net = tf.keras.layers.Dense(10)
#net.bias   # AttributeError: 'Dense' object has no attribute 'bias'

net.get_weights()   # []
net.weights   # []

net.build(input_shape=[None, 4])
net.kernel.shape, net.bias.shape   # (4, 10) (10,)

net.build(input_shape=[None, 20])
net.kernel.shape, net.bias.shape   # (20, 10) (10,)

net.build(input_shape=[2, 4])
net.kernel.shape, net.bias.shape   # (4, 10) (10,)

net.build(input_shape=[None, 20])
net.kernel.shape, net.bias.shape   # (20, 10) (10,)

# out = net(tf.random.normal([4, 12]))   # InvalidArgumentError: Matrix size-incompatible: In[0]: [4,12], In[1]: [20,10] [Op:MatMul]

out = net(tf.random.normal([4, 20]))   # (20, 10) (10,)
out.shape   # (4, 10)

多层全连接层

• ▪ keras.Sequential([layer1, layer2, layer3])

x = tf.random.normal([2, 3])

model = keras.Sequential([
    keras.layers.Dense(2, activation='relu'),
    keras.layers.Dense(2, activation='relu'),
    keras.layers.Dense(2)
])

model.build(input_shape=[2, 3])
model.summary()

for p in model.trainable_weights:
    print(p.name, p.shape)


out = model(x)
print(out.shape)   # (2, 2)

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_2 (Dense)              multiple                  8         
_________________________________________________________________
dense_3 (Dense)              multiple                  6         
_________________________________________________________________
dense_4 (Dense)              multiple                  6         
=================================================================
Total params: 20
Trainable params: 20
Non-trainable params: 0
_________________________________________________________________
dense_2/kernel:0 (3, 2)
dense_2/bias:0 (2,)
dense_3/kernel:0 (2, 2)
dense_3/bias:0 (2,)
dense_4/kernel:0 (2, 2)
dense_4/bias:0 (2,)
(2, 2)