py3: Work with proper native string paths in crypto meta

Code Issues Proposed changes

Previously, we would work with these paths as WSGI strings -- this would
work fine when all data were read and written on the same major version
of Python, but fail pretty badly during and after upgrading Python.
In particular, if a py3 proxy-server tried to read existing data that
was written down by a py2 proxy-server, it would hit an error and
respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
was freshly-written by a py3 proxy, it would serve corrupt data back to
the client (including a corrupt/invalid ETag and Content-Type).
Now, ensure that both py2 and py3 write down paths as native strings.
Make an effort to still work with WSGI-string metadata, though it can be
ambiguous as to whether a string is a WSGI string or not. The heuristic
used is if
 * the path from metadata does not match the (native-string) request
 path and
 * the path from metadata (when interpreted as a WSGI string) can be
 "un-wsgi-fied" without any encode/decode errors and
 * the native-string path from metadata *does* match the native-string
 request path
then trust the path from the request. By contrast, we usually prefer the
path from metadata in case there was a pipeline misconfiguration (see
related bug).
Add the ability to read and write a new, unambiguous version of metadata
that always has the path as a native string. To support rolling
upgrades, a new config option is added: meta_version_to_write. This
defaults to 2 to support rolling upgrades without configuration changes,
but the default may change to 3 in a future release.
UpgradeImpact
=============
When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set
 meta_version_to_write = 1
in your keymaster's configuration. Regardless of prior Swift version, set
 meta_version_to_write = 3
after upgrading all proxy servers.
When switching from Python 2 to Python 3, first upgrade Swift while on
Python 2, then upgrade to Python 3.
Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
Closes-Bug: #1888037
Related-Bug: #1813725

This commit is contained in:

Tim Burke

2020年07月20日 14:18:33 -07:00

parent d6399b32e7

commit 7d429318dd

4 changed files with 310 additions and 17 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

27
etc/keymaster.conf-sample

View File

@@ -1,4 +1,13 @@

[keymaster]

# Over time, the format of crypto metadata on disk may change slightly to resolve

# ambiguities. In general, you want to be writing the newest version, but to

# ensure that all writes can still be read during rolling upgrades, there's the

# option to write older formats as well.

# Before upgrading from Swift 2.20.0 or earlier, ensure this is set to 1

# Before upgrading from Swift 2.25.0 or earlier, ensure this is set to at most 2

# After upgrading all proxy servers, set this to 3 (currently the highest version)

# meta_version_to_write = 3

# Sets the root secret from which encryption keys are derived. This must be set

# before first use to a value that is a base64 encoding of at least 32 bytes.

# The security of all encrypted data critically depends on this key, therefore

@@ -16,6 +25,15 @@

# backends that use Keystone for authentication. Currently, the only

# implemented backend is for Barbican.

# Over time, the format of crypto metadata on disk may change slightly to resolve

# ambiguities. In general, you want to be writing the newest version, but to

# ensure that all writes can still be read during rolling upgrades, there's the

# option to write older formats as well.

# Before upgrading from Swift 2.20.0 or earlier, ensure this is set to 1

# Before upgrading from Swift 2.25.0 or earlier, ensure this is set to at most 2

# After upgrading all proxy servers, set this to 3 (currently the highest version)

# meta_version_to_write = 3

# The api_class tells Castellan which key manager to use to access the external

# key management system. The default value that accesses Barbican is

# castellan.key_manager.barbican_key_manager.BarbicanKeyManager.

@@ -79,6 +97,15 @@

# The kmip_keymaster section is used to configure a keymaster that fetches an

# encryption root secret from a KMIP service.

# Over time, the format of crypto metadata on disk may change slightly to resolve

# ambiguities. In general, you want to be writing the newest version, but to

# ensure that all writes can still be read during rolling upgrades, there's the

# option to write older formats as well.

# Before upgrading from Swift 2.20.0 or earlier, ensure this is set to 1

# Before upgrading from Swift 2.25.0 or earlier, ensure this is set to at most 2

# After upgrading all proxy servers, set this to 3 (currently the highest version)

# meta_version_to_write = 3

# The value of the ``key_id`` option should be the unique identifier for a

# secret that will be retrieved from the KMIP service. The secret should be an

# AES-256 symmetric key.

12
etc/proxy-server.conf-sample

View File

@@ -1113,6 +1113,18 @@ use = egg:swift#copy

[filter:keymaster]

use = egg:swift#keymaster

# Over time, the format of crypto metadata on disk may change slightly to resolve

# ambiguities. In general, you want to be writing the newest version, but to

# ensure that all writes can still be read during rolling upgrades, there's the

# option to write older formats as well.

# Before upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, ensure this is set to 1

# Before upgrading from Swift 2.25.0 or earlier, ensure this is set to at most 2

# After upgrading all proxy servers, set this to 3 (currently the highest version)

#

# The default is currently 2 to support upgrades with no configuration changes,

# but may change to 3 in the future.

meta_version_to_write = 2

# Sets the root secret from which encryption keys are derived. This must be set

# before first use to a value that is a base64 encoding of at least 32 bytes.

# The security of all encrypted data critically depends on this key, therefore