save tuple of simple data types to disk (low memory foot print)

Sat Oct 29 12:47:57 EDT 2011

On 10/29/2011 01:08 AM, Roy Smith wrote:
> In article <mailman.2293.1319834877.27778.python-list at python.org>,
> Gelonida N <gelonida at gmail.com> wrote:
>>> I would like to save many dicts with a fixed amount of keys
>> tuples to a file in a memory efficient manner (no random, but only
>> sequential access is required)
>> There's two possible scenarios here. One, which you seem to be 
> exploring, is to carefully study your data and figure out the best way 
> to externalize it which reduces volume.
>> The other is to just write it out in whatever form is most convenient 
> (JSON is a reasonable thing to try first), and compress the output. Let 
> the compression algorithms worry about extracting the entropy. You may 
> be surprised at how well it works. It's also an easy experiment to try, 
> so if it doesn't work well, at least it didn't cost you much to find out.

Yes I have to make some more tests to see the defference between
just compressing aplain format (JSON / pickle) and compressing the
'optimized' representation.