Scrapy 1.1 Release with Official Python 3 Support

Fasten your seat belts, ladies and gentlemen: Scrapy 1.1 with Python 3 support is officially out! After a couple months of hard work and four release candidates , this is the first official Scrapy release to support Python 3.

We know that many of you have been eagerly looking forward to moving your whole stack to Python 3. Well, wait no more, you can get rid of Python 2 once and for all (for the most part)!

Without further ado, let’s dive into the nuts and bolts of this latest step forward.

What’s new?

Python 3 support isn’t the only good news coming from this release. There are a few features and general improvements that you might want to be aware of:

Item Loaders now support nested loaders

`response.text` now holds the response body as unicode, while `response.body` holds the byte string version

`FormRequest.from_response` now accepts two new arguments: `formcss` and `formid`

Limitations using Python 3

Scrapy on Python 3 doesn’t work in Windows environments yet. Scrapy depends on Twisted and some parts of Twisted haven’t been ported yet. Once this Twisted issue is solved, Scrapy users with Windows will be able to run their spiders on Python 3.

In addition to this, there are a couple of features that are not supported on Python 3:

Backwards incompatible changes

Now when you upload files or images to S3, Scrapy sets them as private instead of public. You can change this behaviour via the FILES_STORE_S3_ACL setting.

FilesPipeline and ImagesPipeline settings are now instance attributes instead of class attributes. There’s a work in progress to solve this. ( PR #1989 )

canonicalize_url() has been changed (for the better). However, it could invalidate some HTTP cache entries you may have from pre-1.1 Scrapy crawls and break some link extractors as well.

A big Thank You to…

We couldn’t have done it without the help of all you folks reporting and fixing issues, requesting and submitting features, commenting on pull requests, improving documentation, etc., the list goes on.

Scrapy is a community and Scrapy 1.1 is the result of this community effort. You should be proud of yourselves. Kudos to all you Scrapy lovers!

We’d also like to nominate and thank everyone (in alphabetical order) who contributed directly to the Scrapy 1.1 source code:

Wrap Up

Python 3 support has been out in beta release for just a few months. Chances are that there are still some corner cases that have yet to be discovered. If you happen to face any unexpected behaviour, please report your findings in the issue tracker .

You can also contribute to the Scrapy community in several ways, such as improving documentation, writing tutorials, fixing bugs and including new features in Scrapy. Check the Contributions Guideline if you want to engage with this amazing community.