Python Pickle: Painless binary storage for Python objects
The pickle module provided with Python is so useful that I’m surprised I haven’t used it before. Pickle allows you to save an entire data structure (such as an object) to disk as a binary file in a effortless (and fairly efficient) manner. For example, in my latest project I have created a Monte Carlo simulation that can take quite a bit of time to run. I also need to make multiple runs to get statistics on the results. At the end of each run, I need to dump the resulting data to disk so that it can be read in later by an analysis program. If I had to write data in a format that could be interchanged with other scientific software, I’d use the hdf5 format with the pytables package. However, right now I just need to get something working, and the pickle module is perfect. Here is how I save an object called box:
pickleFileName = "Pickles/boxData_run" + str(run) + ".pickle"
pickleFile = open(pickleFileName, 'wb')
pickle.dump(box, pickleFile, pickle.HIGHEST_PROTOCOL)
pickleFile.close()First, open a file object for binary writing. Then use pickle.dump() to write the object to a pickle file. That’s all! To read in a pickled object, do the following:
pickleFileName = "Pickles/boxData_run.pickle"
pickleFile = open(pickleFileName, 'rb')
data = pickle.load(pickleFile)
pickleFile.close()The object is “reconstituted” exactly as it was saved–with methods, class and object data, etc. This probably isn’t ideal for a “finished” application written in Python, but it sure works well for the quick and dirty apps that I am always writing.
Comments
Comment #1 by The Python configparser: a way to read simple data files | as through a mirror dimly
The Python configparser: a way to read simple data files | as through a mirror dimly - Feb 2, 2009
[…] some cases, I just hard-code the values in the script that calls the library, and in other cases I load a pickle file containing a Python objext. What if I want to read in data or configuration parameters from a […]
Comment #2 by Storing large Numpy arrays on disk: Python Pickle vs. HDF5adsf | shocksolution.com: scientific computing, modeling, and simulation
[…] In a previous post, I described how Python’s Pickle module is fast and convenient for storing all sorts of data on disk. More recently, I showed how to profile the memory usage of Python code. In recent weeks, I’ve uncovered a serious limitation in the Pickle module when storing large amounts of data: Pickle requires a large amount of memory to save a data structure to disk. Fortunately, there is an open standard called HDF, which defines a binary file format that is designed to efficiently store large scientific data sets. I will demonstrate both approaches, and profile them to see how much memory is required. I am writing the HDF file using the PyTables interface. Here’s the little test program I’ve been using: […]
Comment #3 by jaimearangJaime Arango
jaimearangJaime Arango - Jun 5, 2013
Thank you very much! I wade trough several help files of python looking for this…
Comment #4 by Storing large Numpy arrays on disk: Python Pickle vs. HDF5 – Shocksolution.com
Storing large Numpy arrays on disk: Python Pickle vs. HDF5 – Shocksolution.com - May 6, 2018
[…] In a previous post, I described how Python’s Pickle module is fast and convenient for storing all sorts of data on disk. More recently, I showed how to profile the memory usage of Python code. In recent weeks, I’ve uncovered a serious limitation in the Pickle module when storing large amounts of data: Pickle requires a large amount of memory to save a data structure to disk. Fortunately, there is an open standard called HDF, which defines a binary file format that is designed to efficiently store large scientific data sets. I will demonstrate both approaches, and profile them to see how much memory is required. I am writing the HDF file using the PyTables interface. Here’s the little test program I’ve been using: […]