我正在尝试tf.data.Dataset.group_by_window()
在我拥有的数据集上运行petastorm
make_tf_dataset
并不断收到错误
ValueError: Invalid `key_func`. `key_func` must return a single `tf.int64` scalar tensor but its return type is TensorSpec(shape=(None,), dtype=tf.int64, name=None).
编码:
with test_converter.make_tf_dataset(batch_size=BATCH_SIZE, num_epochs=1) as test_dataset:
tf_test = test_dataset.map(row_generator, num_parallel_calls=tf.data.AUTOTUNE, deterministic=False)
key_func = lambda x: x["my_id_int"]
reduce_func = lambda key, dataset: dataset.batch(100)
tf_test_grp = tf_test.group_by_window(
key_func=key_func, reduce_func=reduce_func, window_size=100)
行生成器是
def row_generator(x):
d = {'my_id_int':x.my_id_int, ...}
return d
和 test_converter 是
test_converter = make_spark_converter(df_test.select(all_fields))
任何人都知道如何解决它?键或映射应该有不同的值吗?