DL笔记:用 python 实现梯度下降的算法

回顾上回讲的梯度下降算法,想实现梯度下降,需要不断更新 w:

$$ \Delta w_{ij} = \eta \delta_j x_i $$

具体步骤如下:

  • 初始化权重变化率为 0 :$\Delta w_i = 0$
  • 对训练集中的每一个数据:
    • 做正前传播计算:$\hat y=f(\sum_iw_ix_i)$
    • 计算输出单元的 error term:$\delta=(y-\hat y) * f’(\sum_iw_ix_i)$
    • 更新权重变化率:$\Delta w_i= \Delta w_i + \delta x_i$
  • 更新权重 $w_i = w_i + \eta \Delta w_i /m$
  • 重复 e 次训练 epochs

代码实现

  1. 初始化权重变化率为 0
1
del_w = np.zeros(weights.shape)
  1. 正向传播计算
1
output = sigmoid(np.dot(x, weights))
  1. 计算输出单元的 error term
1
2
error = y - output
error_term = error * output * (1-output)
  1. 更新权重变化率
1
del_w += error_term * x
  1. 更新权重
1
weights += learnrate * del_w / n_records
  1. 重复 epochs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
for e in range(epochs):
del_w = np.zeros(weights.shape)
for x, y in zip(features.values, targets):
# Loop through all records, x is the input, y is the target

# Calculate the output
output = sigmoid(np.dot(x, weights))

# Calculate the error
error = y - output

# Calculate the error term
error_term = error * output * (1-output)

# Calculate the change in weights for this sample
# and add it to the total weight change
del_w += error_term * x

# Update weights using the learning rate and the average change in weights
weights += learnrate * del_w / n_records

完整代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
import numpy as np
from data_prep import features, targets, features_test, targets_test

# Defining the sigmoid function for activations
def sigmoid(x):
return 1 / (1 + np.exp(-x))

# reserve seed
np.random.seed(42)

n_records, n_features = features.shape
last_loss = None

# Initialize weights
weights = np.random.normal(scale=1 / n_features**.5, size=n_features)

# Neural Network hyperparameters
epochs = 1000
learnrate = 0.5

for e in range(epochs):
del_w = np.zeros(weights.shape)
for x, y in zip(features.values, targets):
# Loop through all records, x is the input, y is the target

# Calculate the output
output = sigmoid(np.dot(x, weights))

# Calculate the error
error = y - output

# Calculate the error term
error_term = error * output * (1-output)

# Calculate the change in weights for this sample
# and add it to the total weight change
del_w += error_term * x

# Update weights using the learning rate and the average change in weights
weights += learnrate * del_w / n_records

# Printing out the mean square error on the training set
if e % (epochs / 10) == 0:
out = sigmoid(np.dot(features, weights))
loss = np.mean((out - targets) ** 2)
if last_loss and last_loss < loss:
print("Train loss: ", loss, " WARNING - Loss Increasing")
else:
print("Train loss: ", loss)
last_loss = loss

# Calculate accuracy on test data
test_out = sigmoid(np.dot(features_test, weights))
predictions = test_out > 0.5
accuracy = np.mean(predictions == targets_test)
print("Prediction accuracy: {:.3f}".format(accuracy))

00 的 DeepLearning 笔记

kidult00 wechat
扫码关注 00 的公众号
如果文章帮您节省时间或者解答疑问,不妨打个赏 :)