Scrapy, write the first crawl problem summary

Because of python's outstanding performance in artificial intelligence, I have been exploring Python recently. I hope to have a step-by-step understanding of Python in the future. I also look forward to discussing python's various issues with you.With a weekend break at home, I wrote a small crawler program, and the problems I encountered during the crawler process are summarized here.

During the installation of Scrapy, I was prompted that the installation failed because I did not have permission to get files, so I switched to superuser installation, which is to add sudo before pip.

(3) Create Scrapy Project

After installing Scrapy, you can create a project using the command line tool "scrapy start project projectname", which is a global command and does not need to be run within the project.The code is as follows:

This problem was encountered because the default python compiler for the ubuntu system is version 2.7. After searching, it was found that the default encoding for python 2.x is ascii, and the code may be caused by UTF-8 characters. The solution is to set utf-8.

Find the file in error and add the following lines after import as follows: