To enable persistence support you just need to define a job directory through
the JOBDIR setting. This directory will be for storing all required data to
keep the state of a single job (ie. a spider run). It’s important to note that
this directory must not be shared by different spiders, or even different
jobs/runs of the same spider, as it’s meant to be used for storing the state of
a single job.

Sometimes you’ll want to keep some persistent spider state between pause/resume
batches. You can use the spider.state attribute for that, which should be a
dict. There’s a built-in extension that takes care of serializing, storing and
loading that attribute from the job directory, when the spider starts and
stops.

Here’s an example of a callback that uses the spider state (other spider code
is omitted for brevity):