Data serialized using the pickle module must be portable across Python
versions. It should also support the latest language features as well
as implementation-specific features. For this reason, the pickle
module knows about several protocols (currently numbered from 0 to 3),
each of which appeared in a different Python version. Using a
low-numbered protocol version allows to exchange data with old Python
versions, while using a high-numbered protocol allows access to newer
features and sometimes more efficient resource use (both CPU time
required for (de)serializing, and disk size / network bandwidth
required for data transfer).

The latest current protocol, coincidentally named protocol 3, appeared
with Python 3.0 and supports the new incompatible features in the
language (mainly, unicode strings by default and the new bytes
object). The opportunity was not taken at the time to improve the
protocol in other ways.

This PEP is an attempt to foster a number of incremental improvements
in a new pickle protocol version. The PEP process is used in order to
gather as many improvements as possible, because the introduction of a
new pickle protocol should be a rare occurrence.

Traditionally, when unpickling an object from a stream (by calling
load() rather than loads()), many small read()
calls can be issued on the file-like object, with a potentially huge
performance impact.

Protocol 4, by contrast, features binary framing. The general structure
of a pickle is thus the following:

To keep the implementation simple, it is forbidden for a pickle opcode
to straddle frame boundaries. The pickler takes care not to produce such
pickles, and the unpickler refuses them. Also, there is no "last frame"
marker. The last frame is simply the one which ends with a STOP opcode.

How the pickler decides to partition the pickle stream into frames is an
implementation detail. For example, "closing" a frame as soon as it
reaches ~64 KiB is a reasonable choice for both performance and pickle
size overhead.

The GLOBAL opcode, which is still used in protocol 3, uses the
so-called "text" mode of the pickle protocol, which involves looking
for newlines in the pickle stream. It also complicates the implementation
of binary framing.

Protocol 4 forbids use of the GLOBAL opcode and replaces it with
GLOBAL_STACK, a new opcode which takes its operand from the stack.

By default, pickle is only able to serialize module-global functions and
classes. Supporting other kinds of objects, such as unbound methods [4],
is a common request. Actually, third-party support for some of them, such
as bound methods, is implemented in the multiprocessing module [5].

The __qualname__ attribute from PEP 3155 makes it possible to
lookup many more objects by name. Making the GLOBAL_STACK opcode accept
dot-separated names would allow the standard pickle implementation to
support all those kinds of objects.

Current protocol versions export object sizes for various built-in
types (str, bytes) as 32-bit ints. This forbids serialization of
large data [1]. New opcodes are required to support very large bytes
and str objects.

Many common built-in types (such as str, bytes, dict, list, tuple)
have dedicated opcodes to improve resource consumption when
serializing and deserializing them; however, sets and frozensets
don't. Adding such opcodes would be an obvious improvement. Also,
dedicated set support could help remove the current impossibility of
pickling self-referential sets [2].

Currently, classes whose __new__ mandates the use of keyword-only
arguments can not be pickled (or, rather, unpickled) [3]. Both a new
special method (__getnewargs_ex__) and a new opcode (NEWOBJ_EX)
are needed. The __getnewargs_ex__ method, if it exists, must
return a two-tuple (args, kwargs) where the first item is the
tuple of positional arguments and the second item is the dict of
keyword arguments for the class's __new__ method.

The PUT opcodes all require an explicit index to select in which entry
of the memo dictionary the top-of-stack is memoized. However, in practice
those numbers are allocated in sequential order. A new opcode, MEMOIZE,
will instead store the top-of-stack in at the index equal to the current
size of the memo dictionary. This allows for shorter pickles, since PUT
opcodes are emitted for all non-atomic datatypes.

Serhiy Storchaka suggested to replace framing with a special PREFETCH
opcode (with a 2- or 4-bytes argument) to declare known pickle chunks
explicitly. Large data may be pickled outside such chunks. A naïve
unpickler should be able to skip the PREFETCH opcode and still decode
pickles properly, but good error handling would require checking that
the PREFETCH length falls on an opcode boundary.