pytorch - RuntimeError: "exp" not implemented for 'torch.LongTensor'

Question

I am following this tutorial: http://nlp.seas.harvard.edu/2018/04/03/attention.html to implement the Transformer model from the "Attention Is All You Need" paper.

However I am getting the following error : RuntimeError: "exp" not implemented for 'torch.LongTensor'

This is the line, in the PositionalEnconding class, that is causing the error:

div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

When it is being constructed here:

pe = PositionalEncoding(20, 0)

Any ideas?? I've already tried converting this to perhaps a Tensor Float type, but this has not worked.

I've even downloaded the whole notebook with accompanying files and the error seems to persist in the original tutorial.

Any ideas what may be causing this error?

Thanks!

Answer 1

我碰巧也跟着这个教程。

对我来说，我刚刚得到了torch.arange生成浮点类型张量

从

position = torch.arange(0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

至

position = torch.arange(0., max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0., d_model, 2) * -(math.log(10000.0) / d_model))

只是一个简单的修复。但现在它对我有用。火炬exp和sin以前的支持有可能LongTensor但现在不再支持（不太确定）。

Answer 2

似乎torch.arange返回 a LongTensor，尝试torch.arange(0.0, d_model, 2)强制 torch 返回 a FloatTensor。

Answer 3

@shai 给出的建议对我有用。我在两个地方修改了使用的init方法：PositionalEncoding0.0

position = torch.arange(0.0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0.0, d_model, 2) * -(math.log(10000.0) / d_model))

Answer 4

就像 Rubens 说的，在更高版本的 Pytorch 中，你不需要担心这些东西。我可以在桌面的 1.8.0 Pytorch 上轻松运行它，但在我的服务器的 1.2.0 Pytorch 中无法通过它。不同版本之间有一些不兼容的地方。

Answer 5

对我来说，安装pytorch == 1.7.1解决了这个问题。