5

I receive an error when running scrapy on Raspberry Pi 3.

I have successfully installed it, but when I try to startproject or crawl with a previously created spider, I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 7, in <module>
    from scrapy.cmdline import execute
  File "/usr/local/lib/python3.4/dist-packages/scrapy/cmdline.py", line 9, in <module>
    from scrapy.crawler import CrawlerProcess
  File "/usr/local/lib/python3.4/dist-packages/scrapy/crawler.py", line 7, in <module>
    from twisted.internet import reactor, defer
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/reactor.py", line 38, in <module>
    from twisted.internet import default
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 56, in <module>
    install = _getInstallFunction(platform)
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 44, in _getInstallFunction
    from twisted.internet.epollreactor import install
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/epollreactor.py", line 24, in <module>
    from twisted.internet import posixbase
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/posixbase.py", line 18, in <module>
    from twisted.internet import error, udp, tcp
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/tcp.py", line 28, in <module>
    from twisted.internet._newtls import (
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_newtls.py", line 21, in <module>
    from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
  File "/usr/local/lib/python3.4/dist-packages/twisted/protocols/tls.py", line 65, in <module>
    from twisted.internet._sslverify import _setAcceptableProtocols
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1865, in <module>
    "ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:"
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1845, in fromOpenSSLCipherString
    SSL.SSLv23_METHOD, SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1797, in _expandCipherString
    ctx.set_cipher_list(cipherString.encode('ascii'))
TypeError: must be str, not bytes

I have no idea why I am getting this or how to fix it, please help?

SOLUTION:

Thanks everyone for the help, I got Scrapy to work on Raspberry Pi 3 in the end by following these steps:

First install virtualenv:

sudo pip install virtualenv

Then create a virtualenv and active it for Scrapy:

virtualenv scrapyenv
source scrapyenv/bin/activate

Then I ran and updated everything in there:

apt-get update
apt-get upgrade

Install all dependencies:

apt-get install libffi-dev
apt-get install libxml2-dev
apt-get install libxslt1-dev
apt-get install libssl-dev
apt-get install python-dev

Then install Scrapy

sudo pip install scrapy

I then updated my pyOpenSSL with this:

pip -vvvv install --upgrade pyOpenSSL

This created a lot of log files and took a bit of time, after that scrapy worked fine with the normal scrapy commands and I have also run a spider - all works.

Improve this question
4
  • 1
    Could you post your spider code? Seems like ssl related issue but the error mentions type issues: "must be str, not bytes" means twisted is expecting python2 instead python3.
    – Granitosaurus
    Commented Feb 2, 2017 at 20:48
  • ok, I wrote my code and tested it in python 3.5, however, I can't even scrapy startproject test without getting the same error as above. So shouldn't be my spider code that is wrong, as I get the same error with just trying to start a project.
    – Svarto
    Commented Feb 2, 2017 at 20:50
  • Are you running it from virtualenvironment? If not try setting one up, installing scrapy there and make sure you have pyOpenSSL installed too
    – Granitosaurus
    Commented Feb 3, 2017 at 5:07
  • Worked with virtualenv and upgrading pyOpenSSL, thanks!
    – Svarto
    Commented Feb 3, 2017 at 8:42

1 Answer 1

Reset to default
0

What user are you using? i think you need use sudo. also update your ssl. looks like the ssl connection has problems..

Have you run : pip install --verbose twisted

Also update your openssl

pip -vvvv install --upgrade pyOpenSSL, please copy the output here to check if is updated

Regards

Improve this answer
6
  • I am running it with user pi. I ran what you suggested, unfortunately it says all requirements already satisfied....
    – Svarto
    Commented Feb 2, 2017 at 20:51
  • have you install this : $ sudo apt-get install libffi-dev $ sudo apt-get install libssl-dev $ sudo apt-get install libxml2-dev libxslt1-dev
    – Jhovanny Uribe
    Commented Feb 2, 2017 at 21:06
  • yes, unfortunately, I also tried running those commands again and for each one it said that I have the newest version already...
    – Svarto
    Commented Feb 2, 2017 at 21:17
  • try updating your openssl, pip -vvvv install --upgrade pyOpenSSL
    – Jhovanny Uribe
    Commented Feb 2, 2017 at 21:25
  • Thanks for the help, it worked by installing a virtualenv and then running the pyOpenSSL upgrade command. This updated my pyOpenSSL and after that it worked - thanks!
    – Svarto
    Commented Feb 3, 2017 at 8:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.