openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Users have complained for a while that Swift's ETags don't match the
expected RFC formats. We've resisted fixing this for just as long,
worrying that the fix would break innumerable clients that expect the
value to be a hex-encoded MD5 digest and *nothing else*.
But, users keep asking for it, and some consumers (including some CDNs)
break if we *don't* have quoted etags -- so, let's make it an option.
With this middleware, Swift users can set metadata per-account or even
per-container to explicitly request RFC compliant etags or not. Swift
operators also get an option to change the default behavior
cluster-wide; it defaults to the old, non-compliant format.
See also:
 - https://tools.ietf.org/html/rfc2616#section-3.11
 - https://tools.ietf.org/html/rfc7232#section-2.3
Closes-Bug: 1099087
Closes-Bug: 1424614
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I380c6e34949d857158e11eb428b3eda9975d855d

This patch adds a new object versioning mode. This new mode provides
a new set of APIs for users to interact with older versions of an
object. It also changes the naming scheme of older versions and adds
a version-id to each object.
This new mode is not backwards compatible or interchangeable with the
other two modes (i.e., stack and history), especially due to the changes
in the namimg scheme of older versions. This new mode will also serve
as a foundation for adding S3 versioning compatibility in the s3api
middleware.
Note that this does not (yet) support using a versioned container as
a source in container-sync. Container sync should be enhanced to sync
previous versions of objects.
Change-Id: Ic7d39ba425ca324eeb4543a2ce8d03428e2225a1
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>

This allows static symlinks to be synced before their target. Dynamic
symlinks could already be synced even if target object has not been
synced, but static links previously required that target object existed
before it can be PUT. Now, have container_sync middleware plumb in an
override like it does for SLO.
Change-Id: I3bfc62b77b247003adcee6bd4d374168bfd6707d

Change-Id: I4cad872ef97313ec19a8e4ef042696565a56d4c5

Apparently this had been done for us previously? See also:
http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012097.html
Change-Id: I27288f9e503f26827a8274d46b07c1330a21de10

Change-Id: I81e5d49623e884def8b5f6cbbf82f9b0d667d64a
Closes-Bug: 1860149

Whitelisted headers include X-Delete-At/X-Delete-After and all Object
metadata headers (X-Object-Meta-*)
Closes-Bug: 1857546
Change-Id: If5fb164693e395f89d57899fb8ab355f1e3f817c

Prior to the related change, clients may have written down X-Delete-At headers
that are outside of the Timestamp range, for example.
Change-Id: Ib8ae7ebcbdb32e0aa58446bd1ef949e5e2f63e74
Related-Change: I23666ec8a067d829eaf9bfe54bd086c320b3429e
Related-Bug: 1821204
Partial-Bug: 1860149

Change-Id: I98ebe15353d675ca00fee387003bf6572ac385e6

Previously, we took the native string filename attribute and put it
directly in the (WSGI string) PATH_INFO field. Now, we convert it to
a WSGI string first.
Change-Id: I30e3beb8707b88c36bd3cdc7a0887d069e943ba6
Closes-Bug: #1858259 

unittest2 was needed for Python version <= 2.6, so it hasn't been needed
for quite some time. See unittest2 note one:
https://docs.python.org/2.7/library/unittest.html
This drops unittest2 in favor of the standard unittest module.
Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

Apparently FakeSwift has always been persnickety about the order of
query params of registered responses and the app making the calls.
Since query params can often be converted to dictionaries the order of
the encoded params should be able to change without effecting the apps
dispatching of registered responses.
Change-Id: Ied68c9334201a7663e9c85f3bdaa5b0643d4b6db

This is to fix the duplicated words issue, like:
 * "both of which which must be stripped off"
 * "In addition the the values set"
 and so on
Change-Id: Id3d84281f15815b4185c76874575e91a3589981b

Change-Id: Ib918f10e95970b9f562b88e923c25608b826b83f

If we move it to constraints it's more globally accessible in our code,
but more importantly it's more obvious to ops that everything breaks if
you try to mis-configure different values per-service.
Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee

An odd thing happened: when my code did only
"from swift.common.ring import Ring", and nothing else,
the interpreter looped on the CPU.
The import from the top of the ring module is our standard
protocol for Ring. It causes no problem in places like
container updater or list_endpoints.py. It is a big
mystery why it causes Python 3.7.3 to loop, and only
in specific circumstances.
But we noticed that the recursive import is unnecesasry
in this case, so an obviously harmless fix exists.
Change-Id: I7373bbb0a50d090d6becf141e5832f8ae98381a4

Previously, if you were on Python 2.7.10+ [0], such a newline would cause the
sharder to fail, complaining about invalid header values when trying to create
the shard containers. On older versions of Python, it would most likely cause a
parsing error in the container-server that was trying to handle the PUT.
Now, quote all places that we pass around container paths. This includes:
 * The X-Container-Sysmeta-Shard-(Quoted-)Root sent when creating the (empty)
 remote shards
 * The X-Container-Sysmeta-Shard-(Quoted-)Root included when initializing the
 local handoff for cleaving
 * The X-Backend-(Quoted-)Container-Path the proxy sends to the object-server
 for container updates
 * The Location header the container-server sends to the object-updater
Note that a new header was required in requests so that servers would
know whether the value should be unquoted or not. We can get away with
reusing Location in responses by having clients opt-in to quoting with
a new X-Backend-Accept-Quoted-Location header.
During a rolling upgrade,
 * old object-servers servicing requests from new proxy-servers will
 not know about the container path override and so will try to update
 the root container,
 * in general, object updates are more likely to land in the root
 container; the sharder will deal with them as misplaced objects, and
 * shard containers created by new code on servers running old code
 will think they are root containers until the server is running new
 code, too; during this time they'll fail the sharder audit and report
 stats to their account, but both of these should get cleared up upon
 upgrade.
Drive-by: fix a "conainer_name" typo that prevented us from testing that
we can shard a container with unicode in its name. Also, add more UTF8
probe tests.
[0] See https://bugs.python.org/issue22928
Change-Id: Ie08f36e31a448a547468dd85911c3a3bc30e89f1
Closes-Bug: 1856894

Since we don't use 404s from handoffs anymore, we need to not let errors
on handoffs overwhelm primary responses either
Change-Id: I2624e113c9d945542f787e5f18f487bd7be3d32e
Closes-Bug: #1857909 

Related-Change: I27701a31bfa22842c272b7781738e8c546b82cbc
Related-Change: If912f71d8b0d03369680374e8233da85d8d38f85
Change-Id: I0c850b8ae40d1ab477d0f5d18f92579c457da54e
Related-Bug: 1475018

Otherwise, we waste a request on some 416/206 response that won't be
helpful.
To do this, add a new X-Backend-Ignore-Range-If-Metadata-Present header
whose value is a comma-separated list of header names. Middlewares may
include this header to tell object-servers to send the whole object
(rather than a 206 or 416) if *any* of the metadata are present.
Have dlo and symlink use it, too; it won't save us any round-trips, but
it should clean up some object-server logging.
Change-Id: I4ff2a178d0456e7e37d561109ef57dd0d92cbd4e

Previously, passing a relative path would confuse the ContainerBroker about
which DB files are available, leading to an IndexError when none were found.
Just call realpath() on whatever the user provided so we don't have to muck
with any of the broker code.
Change-Id: Icdf100dfcd006d975b49d151b99aa9272452d013

The lack of quoting gets extra troublesome with reserved names,
where messages get truncated.
Change-Id: I415901d3a8cd24cb3cedc72235292bb9d1705bbc

Otherwise, we can 500 with
 ValueError: invalid literal for int() with base 10: ''
Change-Id: I35614aa4b42e61d97929579dcb16f7dfc9fef96f

Change-Id: I5b391664b9260c762bd122a92387084c0582a362
Related-Change: I5f3bc6416207cab2c7e3a77ee6689360b55990e7

Change-Id: Ib2df3ec683c1670cadba72d1fc0d666975f64aab

This shouldn't have any practical effect, since *any* False-y value for
chunk will give us the correct behavior, but let's be consistent in the
type used with that name.
Change-Id: I1928883792e7a9945f3731ddeea992d14882d679

... and drive-by a import rename
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I1eaf075ff9855cfa03e7991bdf33375b0e4397e6