Object metadata is stored as a pickled hash: first the data is
pickled, then split into strings of length <= 254, then stored in a
series of extended attributes named "user.swift.metadata",
"user.swift.metadata1", "user.swift.metadata2", and so forth.
The choice of length 254 is odd, undocumented, and dates back to the
initial commit of Swift. From talking to people, I believe this was an
attempt to fit the first xattr in the inode, thus avoiding a
seek. However, it doesn't work. XFS _either_ stores all the xattrs
together in the inode (local), _or_ it spills them all to blocks
located outside the inode (extents or btree). Using short xattrs
actually hurts us here; by splitting into more pieces, we end up with
more names to store, thus reducing the metadata size that'll fit in
the inode.
[Source: http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/Extended_Attributes.html]
I did some benchmarking of read_metadata with various xattr sizes
against an XFS filesystem on a spinning disk, no VMs involved.
Summary:
 name | rank | runs | mean | sd | timesBaseline
------|------|------|-----------|-----------|--------------
32768 | 1 | 2500 | 0.0001195 | 3.75e-05 | 1.0
16384 | 2 | 2500 | 0.0001348 | 1.869e-05 | 1.12809122912
 8192 | 3 | 2500 | 0.0001604 | 2.708e-05 | 1.34210998858
 4096 | 4 | 2500 | 0.0002326 | 0.0004816 | 1.94623473988
 2048 | 5 | 2500 | 0.0003414 | 0.0001409 | 2.85674781189
 1024 | 6 | 2500 | 0.0005457 | 0.0001741 | 4.56648611635
 254 | 7 | 2500 | 0.001848 | 0.001663 | 15.4616067887
Here, "name" is the chunk size for the pickled metadata. A total
metadata size of around 31.5 KiB was used, so the "32768" runs
represent storing everything in one single xattr, while the "254" runs
represent things as they are without this change.
Since bigger xattr chunks make things go faster, the new chunk size is
64 KiB. That's the biggest xattr that XFS allows.
Reading of metadata from existing files is unaffected; the
read_metadata() function already handles xattrs of any size.
On non-XFS filesystems, this is no worse than what came before:
ext4 has a limit of one block (typically 4 KiB) for all xattrs (names
and values) taken together [1], so this change slightly increases the
amount of Swift metadata that can be stored on ext4.
ZFS let me store an xattr with an 8 MiB value, so that's plenty. It'll
probably go further, but I stopped there.
[1] https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes
Change-Id: Ie22db08ac0050eda693de4c30d4bc0d620e7f7d4