key/value store optimized for disk storage

Wed May 2 23:00:28 EDT 2012

On 5/2/2012 10:14 PM, Steve Howell wrote:
> This is slightly off topic, but I'm hoping folks can point me in the
> right direction.
>> I'm looking for a fairly lightweight key/value store that works for
> this type of problem:
>> ideally plays nice with the Python ecosystem
> the data set is static, and written infrequently enough that I
> definitely want *read* performance to trump all
> there is too much data to keep it all in memory (so no memcache)
> users will access keys with fairly uniform, random probability
> the key/value pairs are fairly homogenous in nature:
> keys are<= 16 chars
> values are between 1k and 4k bytes generally
> approx 3 million key/value pairs
> total amount of data == 6Gb
> needs to work on relatively recent versions of FreeBSD and Linux

On my 64bit machine with 64 bit Python, I would consider putting all the 
values in one data file and creating a key:file-offset dict. Each value 
would start with length(value) so that is not needed in memory. The 
dict, once created, could be pickled and unpickled for each run.
-- 
Terry Jan Reedy