1

Python 3 不是我的默认版本。我想使用它,因为我想使用的包toripchanger仅在 Python3 下可用。

所以我的pip3版本是:

C:\Users\Truc>pip3 -V
pip 18.1 from c:\python\python37\lib\site-packages\pip (python 3.7)

我使用了我曾经在 python 2.7 版本中运行的 scrapy.spider,它运行良好,但知道我使用的是 3.7,以便toripchanger在文件的顶部使用它:

#!"C:\Python37\python.exe"
# -*- coding: utf-8 -*-

上面的代码涉及蜘蛛的文件本身,middlewares.pysettings.pyitems.py, 和pipelines.py

middlewares.py我添加了一些行:

...
from scrapy.conf import settings
from toripchanger import TorIpChanger
...
ip_changer = TorIpChanger(reuse_threshold=3)

class ProxyMiddleware(object):
    _requests_count = 0

    def process_request(self, request, spider):
        self._requests_count += 1
        if self._requests_count > 10:
            self._requests_count = 0 
            ip_changer.get_new_ip()

        request.meta['proxy'] = settings.get('HTTP_PROXY')
        spider.log('Proxy : %s' % request.meta['proxy'])

运行蜘蛛时的问题scrapy crawl spider是我收到此错误:

File "C:\Users\Truc\...\middlewares.py", line 13, 
in <module> from toripchanger import TorIpChanger
ImportError: No module named toripchanger

您可能会在本主题末尾看到整行错误

对我来说,问题是我尝试了该主题中提供的内容,所以我pip3 install setuptools什至做了pip3 install --upgrade setuptools,我pip3 install Twisted-18.9.0-cp37-cp37m-win32.whl按照本主题中的建议做了,因为有必要scrapy在我的 python 3.7 版本中安装。

正如我所说,我在文件中c:\python27精确时看到的所有错误行#!"C:\Python37\python.exe"

[in] scrapy crawl spider
[out]

2019-02-18 14:27:11 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot:I_AM_A_POLITE_ROBOT)

2019-02-18 14:27:11 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.5, 
cssselect 1.0.3, parsel 1.4.0, 
w3lib 1.19.0, Twisted 17.9.0, 
Python 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit (Intel)], 
pyOpenSSL 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017), 
cryptography 2.1.4, Platform Windows-7-6.1.7601-SP1
2019-02-18 14:27:11 [scrapy.crawler] INFO: Overridden settings
{'NEWSPIDER_MODULE': 'folder.spiders', 
'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['folder.spiders'],
'RETRY_TIMES': 5, 'BOT_NAME': 'I_AM_A_POLITE_ROBOT',
'RETRY_HTTP_CODES': [401, 403, 404, 408, 500, 502, 503, 504],
'AUTOTHROTTLE_ENABLED': True,'DOWNLOAD_DELAY': 2}

2019-02-18 14:27:11 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.throttle.AutoThrottle']

2019-02-18 14:27:11 [py.warnings] WARNING: 
c:\python27\lib\site-packages\scrapy\utils\deprecate.py:
156:ScrapyDeprecationWarning: 
`scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware` 
class is deprecated, use
`scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`
instead ScrapyDeprecationWarning)
Unhandled error in Deferred:

2019-02-18 14:27:11 [twisted] CRITICAL: Unhandled error in Deferred:

2019-02-18 14:27:11 [twisted] CRITICAL:
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1386, in
_inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\crawler.py", line 98, in crawl
six.reraise(*exc_info)
File "c:\python27\lib\site-packages\scrapy\crawler.py", line 80, in crawl
self.engine = self._create_engine()
File "c:\python27\lib\site-packages\scrapy\crawler.py", line 105, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "c:\python27\lib\site-packages\scrapy\core\engine.py", line 69, in __init__self.downloader = downloader_cls(crawler)
File "c:\python27\lib\site-packages\scrapy\core\downloader\__init__.py", line88, in __init__self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "c:\python27\lib\site-packages\scrapy\middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "c:\python27\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "c:\python27\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object
mod = import_module(module)
File "c:\python27\lib\importlib\__init__.py", line 37, in import_module__import__(name)
File "C:\Users\...\middlewares.py", line 13, in <module>
from toripchanger import TorIpChanger
ImportError: No module named toripchanger
4

2 回答 2

0

好的,万一您在 3.7 之后安装了 Python 2.7,Windows 将默认将最新安装的 Python 版本视为默认版本。

于 2019-02-18T15:37:16.843 回答
0

Windows 将始终使用默认的 python。

https://docs.python.org/3/faq/windows.html#how-do-i-make-python-scripts-executable

要在不更改 Windows 默认值的情况下使用其他 python 版本,您必须执行以下操作(确切路径因系统设置而异):

C:\Program Files\[your python path]\python.exe "myfile.py"
于 2019-02-18T17:18:05.677 回答