Use custom encoding for RingData, not pickle.

Code Issues Proposed changes

Serialize RingData in a versioned, custom format which is a combination
of a JSON-encoded header and .tostring() dumps of the
replica2part2dev_id arrays. This format deserializes hundreds of times
faster than rings serialized with Python 2.7's pickle (a significant
performance regression for ring loading between Python 2.6 and Python
2.7). Fixes bug 1031954.
swift.common.ring.ring.RingData is now responsible for serialization and
deserialization of its data via a new load() class method and save()
object method. The new implementation is backward-compatible; if a ring
does not begin with a new-style magic string, it is assumed to be an
old-style pickle-dumped ring and is handled as before. So new Swift
code can read old rings, but old Swift code will not be able to read
newly-serialized rings. THIS SHOULD BE MENTIONED PROMINENTLY IN THE
RELEASE NOTES.
I didn't want to bite of more than necessary, so I didn't mess with
builder file serialization.
Change-Id: I799b9a4c894d54fb16592443904ac055b2638e2d

This commit is contained in:

Darrell Bishop

2012年08月05日 00:51:49 -07:00

parent ef3f8bb335

commit f8ce43a218

4 changed files with 129 additions and 54 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

15
bin/swift-ring-builder

View File

@@ -610,13 +610,11 @@ swift-ring-builder <builder_file> rebalance

print '-' * 79

status = EXIT_WARNING

ts = time()

pickle.dump(builder.get_ring().to_dict(),

GzipFile(pathjoin(backup_dir, '%d.' % ts +

basename(ring_file)), 'wb'), protocol=2)

builder.get_ring().save(

pathjoin(backup_dir, '%d.' % ts + basename(ring_file)))

pickle.dump(builder.to_dict(), open(pathjoin(backup_dir,

'%d.' % ts + basename(argv[1])), 'wb'), protocol=2)

pickle.dump(builder.get_ring().to_dict(), GzipFile(ring_file, 'wb'),

protocol=2)

builder.get_ring().save(ring_file)

pickle.dump(builder.to_dict(), open(argv[1], 'wb'), protocol=2)

exit(status)

@@ -644,10 +642,9 @@ swift-ring-builder <builder_file> write_ring

'"rebalance"?'

else:

print 'Warning: Writing an empty ring'

pickle.dump(ring_data.to_dict(),

GzipFile(pathjoin(backup_dir, '%d.' % time() +

basename(ring_file)), 'wb'), protocol=2)

pickle.dump(ring_data.to_dict(), GzipFile(ring_file, 'wb'), protocol=2)

ring_data.save(

pathjoin(backup_dir, '%d.' % time() + basename(ring_file)))

ring_data.save(ring_file)

exit(EXIT_SUCCESS)

def pretend_min_part_hours_passed():

Use custom encoding for RingData, not pickle.

15 bin/swift-ring-builder Unescape Escape View File

15
bin/swift-ring-builder

View File