0

假设我有一个自定义层，它使用 TF 2.4 使用外部可训练变量为我计算损失（是的，我知道这是一个愚蠢的例子和损失，它只是为了重现性，实际损失要复杂得多）：

import numpy as np
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from tensorflow.keras.layers import Dense, Layer, Input
from tensorflow.keras import Model
from tensorflow.keras.callbacks import EarlyStopping
import tensorflow as tf

n_col = 10
n_row = 1000
X = np.random.normal(size=(n_row, n_col))
beta = np.arange(10)
y = X @ beta

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

class MyLoss(Layer):
    def __init__(self, var1, var2):
        super(MyLoss, self).__init__()
        self.var1 = tf.Variable(var1)
        self.var2 = tf.Variable(var2)

    def get_vars(self):
        return self.var1, self.var2

    def custom_loss(self, y_true, y_pred):
        return self.var1 ** 2 * tf.math.reduce_mean(tf.math.square(y_true-y_pred)) + self.var2 ** 2

    def call(self, y_true, y_pred):
        self.add_loss(self.custom_loss(y_true, y_pred))
        return y_pred


inputs = Input(shape=(X_train.shape[1],))
y_input = Input(shape=(1,))
hidden1 = Dense(10)(inputs)
output = Dense(1)(hidden1)
my_loss = MyLoss(0.5, 0.5)(y_input, output) # here can also initialize those var1, var2
model = Model(inputs=[inputs, y_input], outputs=my_loss)

model.compile(optimizer= 'adam')

训练这个模型很简单：

history = model.fit([X_train, y_train], None,
                    batch_size=32, epochs=100, validation_split=0.1, verbose=0,
                    callbacks=[EarlyStopping(monitor='val_loss', patience=5)])

如果我们编写一个自定义回调或逐个训练纪元，我们可以看到如何收敛到 0 var1，var2正如预期的那样：

var1_list = []
var2_list = []
for i in range(100):
    if i % 10 == 0:
        print('step %d' % i)
    model.fit([X_train, y_train], None,
              batch_size=32, epochs=1, validation_split=0.1, verbose=0)
    var1, var2 = model.layers[-1].get_vars()
    var1_list.append(var1.numpy())
    var2_list.append(var2.numpy())

plt.plot(var1_list, label='var1')
plt.plot(var2_list, 'r', label='var2')
plt.legend()
plt.show()

在此处输入图像描述

简短的问题：如何根据 和 的收敛性（即它们的向量大小，，并再次假设损失要复杂得多，您不能仅将此向量大小添加到损失中）使模型停止（EarlyStopping与 some ）？patiencevar1var2self.var1**2 + self.var2**2

更长的问题：（如果你有时间/耐心）

  • 是否可以实现自定义Metric并EarlyStopping对其进行跟踪？
  • 在这种情况下，当“收敛”只有“最小值”或“最大值”EarlyStopping时，你将如何专注于“收敛”？mode（我想知道我们可以扩展EarlyStopping而不是扩展Callback）
  • 我们可以在没有指标的情况下使用自定义回调来做到这一点吗？
  • 我们如何结合上面的自定义损失，告诉EarlyStopping注意两者，即“如果你没有看到损失的改善和收敛的改善，耐心=10，就停止”？
4

1 回答 1

2

好吧，至少对于“较短的问题”，这变得非常简单，遵循TF 文档中的这个EarlyStopping示例，并通过关注变量规范的扭曲来实现：

class EarlyStoppingAtVarsConvergence(tf.keras.callbacks.Callback):
    def __init__(self, norm_thresh=0.01, patience=0):
        super(EarlyStoppingAtVarsConvergence, self).__init__()
        self.norm_thresh = norm_thresh
        self.patience = patience

    def on_train_begin(self, logs=None):
        # The number of epoch it has waited when norm hasn't converged.
        self.wait = 0
        # The epoch the training stops at.
        self.stopped_epoch = 0
        # Initialize sigmas norm.
        self.vars_norm = self.get_vars_norm()

    def get_vars_norm(self):
        var1, var2 = model.layers[-1].get_vars()
        return var1**2 + var2**2
    
    def on_epoch_end(self, epoch, logs=None):
        current_norm = self.get_vars_norm()
        if np.abs(current_norm - self.vars_norm) > self.norm_thresh:
            self.sigmas_norm = current_norm
            self.wait = 0
        else:
            self.wait += 1
            if self.wait >= self.patience:
                self.stopped_epoch = epoch
                self.model.stop_training = True

    def on_train_end(self, logs=None):
        if self.stopped_epoch > 0:
            print("Epoch %05d: early stopping" % (self.stopped_epoch + 1))

然后模型将运行：

history = model.fit([X_train, y_train], None,
                    batch_size=32, epochs=100, validation_split=0.1, verbose=0,
                    callbacks=[EarlyStoppingAtVarsConvergence(patience=5)])
于 2021-01-25T10:00:14.893 回答