0

按照Andrew Traks 的示例，我想实现一个 3 层神经网络 - 1 个输入，1 个隐藏，1 个输出 - 带有一个简单的 dropout，用于二进制分类。

如果我包含偏差项b1和b2，那么我需要稍微修改 Andrew 的代码，如下所示。

X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])
y = np.array([[0,1,1,0]]).T
alpha,hidden_dim,dropout_percent = (0.5,4,0.2)
synapse_0 = 2*np.random.random((X.shape[1],hidden_dim)) - 1
synapse_1 = 2*np.random.random((hidden_dim,1)) - 1
b1 = np.zeros(hidden_dim)
b2 = np.zeros(1)
for j in range(60000):
    # sigmoid activation function
    layer_1 = (1/(1+np.exp(-(np.dot(X,synapse_0) + b1))))
    # dropout
    layer_1 *= np.random.binomial([np.ones((len(X),hidden_dim))],1-dropout_percent)[0] * (1.0/(1-dropout_percent))
    layer_2 = 1/(1+np.exp(-(np.dot(layer_1,synapse_1) + b2)))
    # sigmoid derivative = s(x)(1-s(x))
    layer_2_delta = (layer_2 - y)*(layer_2*(1-layer_2))
    layer_1_delta = layer_2_delta.dot(synapse_1.T) * (layer_1 * (1-layer_1))
    synapse_1 -= (alpha * layer_1.T.dot(layer_2_delta))
    synapse_0 -= (alpha * X.T.dot(layer_1_delta))
    b1 -= alpha*layer_1_delta
    b2 -= alpha*layer_2_delta

问题当然是，上面的代码与 的尺寸b1不匹配，与和layer_1_delta类似。b2layer_2_delta

我不明白如何计算 delta 以更新b1和b2- 根据Michael Nielsen 的示例，b1并且b2应该由我认为在我的代码中的 deltalayer_1_delta分别更新layer_2_delta。

我在这里做错了什么？我是否弄乱了增量或偏差的维度？我觉得是后者，因为如果我从这段代码中消除偏见，它就可以正常工作。提前致谢

4

1 回答 1

1

因此，首先我将更改为 0 和 1 以对应于X，因为这是它们所属的地方并且它使得它：bXsynapse_X

b1 -= alpha * 1.0 / m * np.sum(layer_2_delta)
b0 -= alpha * 1.0 / m * np.sum(layer_1_delta)

哪里m是训练集中的示例数。此外，掉率非常高，实际上会伤害收敛。所以总的来说考虑了整个代码：

import numpy as np

X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])
m = X.shape[0]
y = np.array([[0,1,1,0]]).T
alpha,hidden_dim,dropout_percent = (0.5,4,0.02)
synapse_0 = 2*np.random.random((X.shape[1],hidden_dim)) - 1
synapse_1 = 2*np.random.random((hidden_dim,1)) - 1
b0 = np.zeros(hidden_dim)
b1 = np.zeros(1)
for j in range(10000):
    # sigmoid activation function
    layer_1 = (1/(1+np.exp(-(np.dot(X,synapse_0) + b0))))
    # dropout
    layer_1 *= np.random.binomial([np.ones((len(X),hidden_dim))],1-dropout_percent)[0] * (1.0/(1-dropout_percent))
    layer_2 = 1/(1+np.exp(-(np.dot(layer_1,synapse_1) + b1)))
    # sigmoid derivative = s(x)(1-s(x))
    layer_2_delta = (layer_2 - y)*(layer_2*(1-layer_2))
    layer_1_delta = layer_2_delta.dot(synapse_1.T) * (layer_1 * (1-layer_1))
    synapse_1 -= (alpha * layer_1.T.dot(layer_2_delta))
    synapse_0 -= (alpha * X.T.dot(layer_1_delta))
    b1 -= alpha * 1.0 / m * np.sum(layer_2_delta)
    b0 -= alpha * 1.0 / m * np.sum(layer_1_delta)

print layer_2
于 2017-12-07T00:11:20.813 回答