:mod:`pickle` --- Python object serialization

.. module:: pickle
 :synopsis: Convert Python objects to streams of bytes and back.

.. sectionauthor:: Jim Kerr <jbkerr@sr.hp.com>.

.. sectionauthor:: Barry Warsaw <barry@python.org>

.. index::
 single: persistence
 pair: persistent; objects
 pair: serializing; objects
 pair: marshalling; objects
 pair: flattening; objects
 pair: pickling; objects

.. seealso::
 The :mod:`json` module: a standard library module allowing JSON
 serialization and deserialization.

.. index::
 single: External Data Representation

Module Interface

To serialize an object hierarchy, you simply call the :func:`dumps` function. Similarly, to de-serialize a data stream, you call the :func:`loads` function. However, if you want more control over serialization and de-serialization, you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively.

The :mod:`pickle` module provides the following constants:

.. data:: HIGHEST_PROTOCOL

 An integer, the highest :ref:`protocol version <pickle-protocols>`
 available. This value can be passed as a *protocol* value to functions
 :func:`dump` and :func:`dumps` as well as the :class:`Pickler`
 constructor.

.. data:: DEFAULT_PROTOCOL

 An integer, the default :ref:`protocol version <pickle-protocols>` used
 for pickling. May be less than :data:`HIGHEST_PROTOCOL`. Currently the
 default protocol is 3, a new protocol designed for Python 3.

The :mod:`pickle` module provides the following functions to make the pickling process more convenient:

.. function:: dump(obj, file, protocol=None, \*, fix_imports=True)

 Write a pickled representation of *obj* to the open :term:`file object` *file*.
 This is equivalent to ``Pickler(file, protocol).dump(obj)``.

 The optional *protocol* argument, an integer, tells the pickler to use
 the given protocol; supported protocols are 0 to :data:`HIGHEST_PROTOCOL`.
 If not specified, the default is :data:`DEFAULT_PROTOCOL`. If a negative
 number is specified, :data:`HIGHEST_PROTOCOL` is selected.

 The *file* argument must have a write() method that accepts a single bytes
 argument. It can thus be an on-disk file opened for binary writing, an
 :class:`io.BytesIO` instance, or any other custom object that meets this
 interface.

 If *fix_imports* is true and *protocol* is less than 3, pickle will try to
 map the new Python 3 names to the old module names used in Python 2, so
 that the pickle data stream is readable with Python 2.

.. function:: dumps(obj, protocol=None, \*, fix_imports=True)

 Return the pickled representation of the object as a :class:`bytes` object,
 instead of writing it to a file.

 Arguments *protocol* and *fix_imports* have the same meaning as in
 :func:`dump`.

.. function:: load(file, \*, fix_imports=True, encoding="ASCII", errors="strict")

 Read a pickled object representation from the open :term:`file object`
 *file* and return the reconstituted object hierarchy specified therein.
 This is equivalent to ``Unpickler(file).load()``.

 The protocol version of the pickle is detected automatically, so no
 protocol argument is needed. Bytes past the pickled object's
 representation are ignored.

 The argument *file* must have two methods, a read() method that takes an
 integer argument, and a readline() method that requires no arguments. Both
 methods should return bytes. Thus *file* can be an on-disk file opened for
 binary reading, an :class:`io.BytesIO` object, or any other custom object
 that meets this interface.

 Optional keyword arguments are *fix_imports*, *encoding* and *errors*,
 which are used to control compatibility support for pickle stream generated
 by Python 2. If *fix_imports* is true, pickle will try to map the old
 Python 2 names to the new names used in Python 3. The *encoding* and
 *errors* tell pickle how to decode 8-bit string instances pickled by Python
 2; these default to 'ASCII' and 'strict', respectively. The *encoding* can
 be 'bytes' to read these 8-bit string instances as bytes objects.
 Using ``encoding='latin1'`` is required for unpickling NumPy arrays and
 instances of :class:`~datetime.datetime`, :class:`~datetime.date` and
 :class:`~datetime.time` pickled by Python 2.

.. function:: loads(bytes_object, \*, fix_imports=True, encoding="ASCII", errors="strict")

 Read a pickled object hierarchy from a :class:`bytes` object and return the
 reconstituted object hierarchy specified therein.

 The protocol version of the pickle is detected automatically, so no
 protocol argument is needed. Bytes past the pickled object's
 representation are ignored.

 Optional keyword arguments are *fix_imports*, *encoding* and *errors*,
 which are used to control compatibility support for pickle stream generated
 by Python 2. If *fix_imports* is true, pickle will try to map the old
 Python 2 names to the new names used in Python 3. The *encoding* and
 *errors* tell pickle how to decode 8-bit string instances pickled by Python
 2; these default to 'ASCII' and 'strict', respectively. The *encoding* can
 be 'bytes' to read these 8-bit string instances as bytes objects.
 Using ``encoding='latin1'`` is required for unpickling NumPy arrays and
 instances of :class:`~datetime.datetime`, :class:`~datetime.date` and
 :class:`~datetime.time` pickled by Python 2.

The :mod:`pickle` module defines three exceptions:

.. exception:: PickleError

 Common base class for the other pickling exceptions. It inherits
 :exc:`Exception`.

.. exception:: PicklingError

 Error raised when an unpicklable object is encountered by :class:`Pickler`.
 It inherits :exc:`PickleError`.

 Refer to :ref:`pickle-picklable` to learn what kinds of objects can be
 pickled.

.. exception:: UnpicklingError

 Error raised when there is a problem unpickling an object, such as a data
 corruption or a security violation. It inherits :exc:`PickleError`.

 Note that other exceptions may also be raised during unpickling, including
 (but not necessarily limited to) AttributeError, EOFError, ImportError, and
 IndexError.

The :mod:`pickle` module exports two classes, :class:`Pickler` and :class:`Unpickler`:

What can be pickled and unpickled?

The following types can be pickled:

• None, True, and False
• integers, floating point numbers, complex numbers
• strings, bytes, bytearrays
• tuples, lists, sets, and dictionaries containing only picklable objects
• functions defined at the top level of a module (using :keyword:`def`, not :keyword:`lambda`)
• built-in functions defined at the top level of a module
• classes that are defined at the top level of a module
• instances of such classes whose :attr:`~object.__dict__` or the result of calling :meth:`__getstate__` is picklable (see section :ref:`pickle-inst` for details).

Attempts to pickle unpicklable objects will raise the :exc:`PicklingError` exception; when this happens, an unspecified number of bytes may have already been written to the underlying file. Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a :exc:`RecursionError` will be raised in this case. You can carefully raise this limit with :func:`sys.setrecursionlimit`.

Note that functions (built-in and user-defined) are pickled by "fully qualified" name reference, not by value. [2] This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. [3]

Similarly, classes are pickled by named reference, so the same restrictions in the unpickling environment apply. Note that none of the class's code or data is pickled, so in the following example the class attribute attr is not restored in the unpickling environment:

class Foo:
 attr = 'A class attribute'

picklestring = pickle.dumps(Foo)

These restrictions are why picklable functions and classes must be defined in the top level of a module.

Similarly, when class instances are pickled, their class's code and data are not pickled along with them. Only the instance data are pickled. This is done on purpose, so you can fix bugs in a class or add methods to the class and still load objects that were created with an earlier version of the class. If you plan to have long-lived objects that will see many versions of a class, it may be worthwhile to put a version number in the objects so that suitable conversions can be made by the class's :meth:`__setstate__` method.

Pickling Class Instances

.. currentmodule:: None

In this section, we describe the general mechanisms available to you to define, customize, and control how class instances are pickled and unpickled.

In most cases, no additional code is needed to make instances picklable. By default, pickle will retrieve the class and the attributes of an instance via introspection. When a class instance is unpickled, its :meth:`__init__` method is usually not invoked. The default behaviour first creates an uninitialized instance and then restores the saved attributes. The following code shows an implementation of this behaviour:

def save(obj):
 return (obj.__class__, obj.__dict__)

def load(cls, attributes):
 obj = cls.__new__(cls)
 obj.__dict__.update(attributes)
 return obj

Classes can alter the default behaviour by providing one or several special methods:

.. method:: object.__getnewargs_ex__()

 In protocols 2 and newer, classes that implements the
 :meth:`__getnewargs_ex__` method can dictate the values passed to the
 :meth:`__new__` method upon unpickling. The method must return a pair
 ``(args, kwargs)`` where *args* is a tuple of positional arguments
 and *kwargs* a dictionary of named arguments for constructing the
 object. Those will be passed to the :meth:`__new__` method upon
 unpickling.

 You should implement this method if the :meth:`__new__` method of your
 class requires keyword-only arguments. Otherwise, it is recommended for
 compatibility to implement :meth:`__getnewargs__`.

 .. versionchanged:: 3.6
 :meth:`__getnewargs_ex__` is now used in protocols 2 and 3.

.. method:: object.__getnewargs__()

 This method serves a similar purpose as :meth:`__getnewargs_ex__`, but
 supports only positional arguments. It must return a tuple of arguments
 ``args`` which will be passed to the :meth:`__new__` method upon unpickling.

 :meth:`__getnewargs__` will not be called if :meth:`__getnewargs_ex__` is
 defined.

 .. versionchanged:: 3.6
 Before Python 3.6, :meth:`__getnewargs__` was called instead of
 :meth:`__getnewargs_ex__` in protocols 2 and 3.

.. method:: object.__getstate__()

 Classes can further influence how their instances are pickled; if the class
 defines the method :meth:`__getstate__`, it is called and the returned object
 is pickled as the contents for the instance, instead of the contents of the
 instance's dictionary. If the :meth:`__getstate__` method is absent, the
 instance's :attr:`~object.__dict__` is pickled as usual.

.. method:: object.__setstate__(state)

 Upon unpickling, if the class defines :meth:`__setstate__`, it is called with
 the unpickled state. In that case, there is no requirement for the state
 object to be a dictionary. Otherwise, the pickled state must be a dictionary
 and its items are assigned to the new instance's dictionary.

 .. note::

 If :meth:`__getstate__` returns a false value, the :meth:`__setstate__`
 method will not be called upon unpickling.

Refer to the section :ref:`pickle-state` for more information about how to use the methods :meth:`__getstate__` and :meth:`__setstate__`.

.. index:: pair: copy; protocol

As we shall see, pickle does not use directly the methods described above. In fact, these methods are part of the copy protocol which implements the :meth:`__reduce__` special method. The copy protocol provides a unified interface for retrieving the data necessary for pickling and copying objects. [4]

Although powerful, implementing :meth:`__reduce__` directly in your classes is error prone. For this reason, class designers should use the high-level interface (i.e., :meth:`__getnewargs_ex__`, :meth:`__getstate__` and :meth:`__setstate__`) whenever possible. We will show, however, cases where using :meth:`__reduce__` is the only option or leads to more efficient pickling or both.

.. method:: object.__reduce__()

 The interface is currently defined as follows. The :meth:`__reduce__` method
 takes no argument and shall return either a string or preferably a tuple (the
 returned object is often referred to as the "reduce value").

 If a string is returned, the string should be interpreted as the name of a
 global variable. It should be the object's local name relative to its
 module; the pickle module searches the module namespace to determine the
 object's module. This behaviour is typically useful for singletons.

 When a tuple is returned, it must be between two and five items long.
 Optional items can either be omitted, or ``None`` can be provided as their
 value. The semantics of each item are in order:

 .. XXX Mention __newobj__ special-case?

 * A callable object that will be called to create the initial version of the
 object.

 * A tuple of arguments for the callable object. An empty tuple must be given
 if the callable does not accept any argument.

 * Optionally, the object's state, which will be passed to the object's
 :meth:`__setstate__` method as previously described. If the object has no
 such method then, the value must be a dictionary and it will be added to
 the object's :attr:`~object.__dict__` attribute.

 * Optionally, an iterator (and not a sequence) yielding successive items.
 These items will be appended to the object either using
 ``obj.append(item)`` or, in batch, using ``obj.extend(list_of_items)``.
 This is primarily used for list subclasses, but may be used by other
 classes as long as they have :meth:`append` and :meth:`extend` methods with
 the appropriate signature. (Whether :meth:`append` or :meth:`extend` is
 used depends on which pickle protocol version is used as well as the number
 of items to append, so both must be supported.)

 * Optionally, an iterator (not a sequence) yielding successive key-value
 pairs. These items will be stored to the object using ``obj[key] =
 value``. This is primarily used for dictionary subclasses, but may be used
 by other classes as long as they implement :meth:`__setitem__`.

.. method:: object.__reduce_ex__(protocol)

 Alternatively, a :meth:`__reduce_ex__` method may be defined. The only
 difference is this method should take a single integer argument, the protocol
 version. When defined, pickle will prefer it over the :meth:`__reduce__`
 method. In addition, :meth:`__reduce__` automatically becomes a synonym for
 the extended version. The main use for this method is to provide
 backwards-compatible reduce values for older Python releases.

.. currentmodule:: pickle

Persistence of External Objects

.. index::
 single: persistent_id (pickle protocol)
 single: persistent_load (pickle protocol)

For the benefit of object persistence, the :mod:`pickle` module supports the notion of a reference to an object outside the pickled data stream. Such objects are referenced by a persistent ID, which should be either a string of alphanumeric characters (for protocol 0) [5] or just an arbitrary object (for any newer protocol).

The resolution of such persistent IDs is not defined by the :mod:`pickle` module; it will delegate this resolution to the user defined methods on the pickler and unpickler, :meth:`~Pickler.persistent_id` and :meth:`~Unpickler.persistent_load` respectively.

To pickle objects that have an external persistent id, the pickler must have a custom :meth:`~Pickler.persistent_id` method that takes an object as an argument and returns either None or the persistent id for that object. When None is returned, the pickler simply pickles the object as normal. When a persistent ID string is returned, the pickler will pickle that object, along with a marker so that the unpickler will recognize it as a persistent ID.

To unpickle external objects, the unpickler must have a custom :meth:`~Unpickler.persistent_load` method that takes a persistent ID object and returns the referenced object.

Here is a comprehensive example presenting how persistent ID can be used to pickle external objects by reference.

.. literalinclude:: ../includes/dbpickle.py

Dispatch Tables

If one wants to customize pickling of some classes without disturbing any other code which depends on pickling, then one can create a pickler with a private dispatch table.

The global dispatch table managed by the :mod:`copyreg` module is available as :data:`copyreg.dispatch_table`. Therefore, one may choose to use a modified copy of :data:`copyreg.dispatch_table` as a private dispatch table.

For example

f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass

creates an instance of :class:`pickle.Pickler` with a private dispatch table which handles the SomeClass class specially. Alternatively, the code

class MyPickler(pickle.Pickler):
 dispatch_table = copyreg.dispatch_table.copy()
 dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)

does the same, but all instances of MyPickler will by default share the same dispatch table. The equivalent code using the :mod:`copyreg` module is

copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)

Handling Stateful Objects

.. index::
 single: __getstate__() (copy protocol)
 single: __setstate__() (copy protocol)

Here's an example that shows how to modify pickling behavior for a class. The :class:`TextReader` class opens a text file, and returns the line number and line contents each time its :meth:`!readline` method is called. If a :class:`TextReader` instance is pickled, all attributes except the file object member are saved. When the instance is unpickled, the file is reopened, and reading resumes from the last location. The :meth:`__setstate__` and :meth:`__getstate__` methods are used to implement this behavior.

class TextReader:
 """Print and number lines in a text file."""

 def __init__(self, filename):
 self.filename = filename
 self.file = open(filename)
 self.lineno = 0

 def readline(self):
 self.lineno += 1
 line = self.file.readline()
 if not line:
 return None
 if line.endswith('\n'):
 line = line[:-1]
 return "%i: %s" % (self.lineno, line)

 def __getstate__(self):
 # Copy the object's state from self.__dict__ which contains
 # all our instance attributes. Always use the dict.copy()
 # method to avoid modifying the original state.
 state = self.__dict__.copy()
 # Remove the unpicklable entries.
 del state['file']
 return state

 def __setstate__(self, state):
 # Restore instance attributes (i.e., filename and lineno).
 self.__dict__.update(state)
 # Restore the previously opened file's state. To do so, we need to
 # reopen it and read from it until the line count is restored.
 file = open(self.filename)
 for _ in range(self.lineno):
 file.readline()
 # Finally, save the file.
 self.file = file

A sample usage might be something like this:

>>> reader = TextReader("hello.txt")
>>> reader.readline()
'1: Hello world!'
>>> reader.readline()
'2: I am line number two.'
>>> new_reader = pickle.loads(pickle.dumps(reader))
>>> new_reader.readline()
'3: Goodbye!'

Restricting Globals

.. index::
 single: find_class() (pickle protocol)

By default, unpickling will import any class or function that it finds in the pickle data. For many applications, this behaviour is unacceptable as it permits the unpickler to import and invoke arbitrary code. Just consider what this hand-crafted pickle data stream does when loaded:

>>> import pickle
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
hello world
0

In this example, the unpickler imports the :func:`os.system` function and then apply the string argument "echo hello world". Although this example is inoffensive, it is not difficult to imagine one that could damage your system.

For this reason, you may want to control what gets unpickled by customizing :meth:`Unpickler.find_class`. Unlike its name suggests, :meth:`Unpickler.find_class` is called whenever a global (i.e., a class or a function) is requested. Thus it is possible to either completely forbid globals or restrict them to a safe subset.

Here is an example of an unpickler allowing only few safe classes from the :mod:`builtins` module to be loaded:

import builtins
import io
import pickle

safe_builtins = {
 'range',
 'complex',
 'set',
 'frozenset',
 'slice',
}

class RestrictedUnpickler(pickle.Unpickler):

 def find_class(self, module, name):
 # Only allow safe classes from builtins.
 if module == "builtins" and name in safe_builtins:
 return getattr(builtins, name)
 # Forbid everything else.
 raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
 (module, name))

def restricted_loads(s):
 """Helper function analogous to pickle.loads()."""
 return RestrictedUnpickler(io.BytesIO(s)).load()

A sample usage of our unpickler working has intended:

>>> restricted_loads(pickle.dumps([1, 2, range(15)]))
[1, 2, range(0, 15)]
>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
Traceback (most recent call last):
 ...
pickle.UnpicklingError: global 'os.system' is forbidden
>>> restricted_loads(b'cbuiltins\neval\n'
... b'(S\'getattr(__import__("os"), "system")'
... b'("echo hello world")\'\ntR.')
Traceback (most recent call last):
 ...
pickle.UnpicklingError: global 'builtins.eval' is forbidden

As our examples shows, you have to be careful with what you allow to be unpickled. Therefore if security is a concern, you may want to consider alternatives such as the marshalling API in :mod:`xmlrpc.client` or third-party solutions.

Performance

Recent versions of the pickle protocol (from protocol 2 and upwards) feature efficient binary encodings for several common features and built-in types. Also, the :mod:`pickle` module has a transparent optimizer written in C.

Examples

For the simplest code, use the :func:`dump` and :func:`load` functions.

import pickle

# An arbitrary collection of objects supported by pickle.
data = {
 'a': [1, 2.0, 3, 4+6j],
 'b': ("character string", b"byte string"),
 'c': {None, True, False}
}

with open('data.pickle', 'wb') as f:
 # Pickle the 'data' dictionary using the highest protocol available.
 pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)

The following example reads the resulting pickled data.

import pickle

with open('data.pickle', 'rb') as f:
 # The protocol version used is detected automatically, so we do not
 # have to specify it.
 data = pickle.load(f)

.. seealso::

 Module :mod:`copyreg`
 Pickle interface constructor registration for extension types.

 Module :mod:`pickletools`
 Tools for working with and analyzing pickled data.

 Module :mod:`shelve`
 Indexed databases of objects; uses :mod:`pickle`.

 Module :mod:`copy`
 Shallow and deep object copying.

 Module :mod:`marshal`
 High-performance serialization of built-in types.

Footnotes

此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。

如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。