Python read/write pickled file

Question

Is there a way in Python 2.7 to improve reading/writing speed (or memory consumption of the file) compared to this version?

import gzip
import cPickle
import io
# save zipped and pickled file
def save_zipped_pickle(obj, filename):
 # disable garbage collector (hack for faster reading/writing)
 gc.disable()
 with gzip.open(filename, 'wb') as f:
 cPickle.dump(obj, io.BufferedWriter(f), -1)
 # enable garbage collector again
 gc.enable()
# load zipped and pickled file
def load_zipped_pickle(filename):
 # disable garbage collector (hack for faster reading/writing)
 gc.disable()
 with gzip.open(filename, 'rb') as f:
 loaded_object = cPickle.load(io.BufferedReader(f))
 # enable garbage collector again
 gc.enable()
 return loaded_object

Answer 1

This looks really efficient and potentially the best you can do, but here are some thoughts:

• try specifying the highest protocol (for Python 2.7 that would be protocol 2):

  with gzip.open(filename, 'wb') as f:
   cPickle.dump(obj, io.BufferedWriter(f), protocol=cPickle.HIGHEST_PROTOCOL)
  

• re-iterate over the serialization format itself - if JSON is an option, ujson might win the performance battle against cPickle

• see if PyPy is an option

• you can also get Cython into play here - there are famous stories how just specifying types with Cython leads to serious performance gains
• there is also this undocumented .fast option/flag, but I am not at all sure it would actually help to speed things up - too good to be true :)
• as far as improving on memory, you can try chunking with shutil

FYI, the gc.disable/gc.enable() pair is planned to be implemented as a context manager in Python 3.7. You can borrow a Python sample from the issue tracker:

@contextmanager
def gc_disabled():
 if gc.isenabled():
 gc.disable()
 yield
 gc.enable()
 else:
 yield