openstack/swift - swift - OpenDev: Free Software Needs Free Tools

When client sends a '?multipart-manifest=get&format=raw' request
middleware will change the manifest returned from object server.
This patch makes sure the response etag is updated to reflect
changes to manifest content
Change-Id: I0ac6dd0808fb041ba7663f4a472a06ee3f1d9a71

Change-Id: I60d4ad62c7d14ed15e114ed95102a2368c48f9f4

Change-Id: Ifcb5013a5728d93cf5491fbff81b2677450698e6

Drive-by: always use quote=True; it basically never hurts.
Change-Id: Id91fdd19f226e9ec0d9c702d40d041c385c52b88

Translate AWS S3 Object Versioning API requests to native Swift Object
Versioning API, speficially:
 * bucket versioning status
 * bucket versioned objects listing params
 * object GETorHEAD & DELETE versionId
 * multi_delete versionId
Change-Id: I8296681b61996e073b3ba12ad46f99042dc15c37
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

Specify that the relink command must be run as the same user than the
daemon processes to avoid creating files that are not manipulable by the
server/replicator/...
Change-Id: If23592855db5f5bb0ec1e7c679de15769fd86871

Change-Id: I331b6871e5b62f61809338a1abddafe1263e7f02

It helps an operator to understand why an SSYNC replication is not
progressing on a specific partition.
Change-Id: I75ccc833d579a98eff22377a83b703fd7e9c4720

Change-Id: I1611e34846e586703e9d3709fa64e8df41f2d685

Users have complained for a while that Swift's ETags don't match the
expected RFC formats. We've resisted fixing this for just as long,
worrying that the fix would break innumerable clients that expect the
value to be a hex-encoded MD5 digest and *nothing else*.
But, users keep asking for it, and some consumers (including some CDNs)
break if we *don't* have quoted etags -- so, let's make it an option.
With this middleware, Swift users can set metadata per-account or even
per-container to explicitly request RFC compliant etags or not. Swift
operators also get an option to change the default behavior
cluster-wide; it defaults to the old, non-compliant format.
See also:
 - https://tools.ietf.org/html/rfc2616#section-3.11
 - https://tools.ietf.org/html/rfc7232#section-2.3
Closes-Bug: 1099087
Closes-Bug: 1424614
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I380c6e34949d857158e11eb428b3eda9975d855d

testtools (required by os-testr) still requires unittest2; until
 * testtools no longer requires unittest2 and
 * we bump up our minimum-supported version of testtools to a version
 that doesn't reference unittest2,
we should continue listing unittest2 in lower-constraints.txt (just in
case they ever have a new release).
Admittedly, that seems unlikely at this point -- unittest2's latest
release was back in 2015.
Change-Id: I09301bf81d68357ec006a21dd37e72c56cefffe3

This patch adds a new object versioning mode. This new mode provides
a new set of APIs for users to interact with older versions of an
object. It also changes the naming scheme of older versions and adds
a version-id to each object.
This new mode is not backwards compatible or interchangeable with the
other two modes (i.e., stack and history), especially due to the changes
in the namimg scheme of older versions. This new mode will also serve
as a foundation for adding S3 versioning compatibility in the s3api
middleware.
Note that this does not (yet) support using a versioned container as
a source in container-sync. Container sync should be enhanced to sync
previous versions of objects.
Change-Id: Ic7d39ba425ca324eeb4543a2ce8d03428e2225a1
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>

This allows static symlinks to be synced before their target. Dynamic
symlinks could already be synced even if target object has not been
synced, but static links previously required that target object existed
before it can be PUT. Now, have container_sync middleware plumb in an
override like it does for SLO.
Change-Id: I3bfc62b77b247003adcee6bd4d374168bfd6707d

Change-Id: I4cad872ef97313ec19a8e4ef042696565a56d4c5

Looking at the last few hundred runs, we've had ~8% TIMEOUTs, and the
average time is already 21 minutes. Let's give it an extra 10 minutes,
see if the TIMEOUT rate comes down.
Change-Id: Ifd9fbac032825bae1c8f5edd88c5d692a0b2cef1

Apparently this had been done for us previously? See also:
http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012097.html
Change-Id: I27288f9e503f26827a8274d46b07c1330a21de10

Change-Id: I81e5d49623e884def8b5f6cbbf82f9b0d667d64a
Closes-Bug: 1860149

Whitelisted headers include X-Delete-At/X-Delete-After and all Object
metadata headers (X-Object-Meta-*)
Closes-Bug: 1857546
Change-Id: If5fb164693e395f89d57899fb8ab355f1e3f817c

Prior to the related change, clients may have written down X-Delete-At headers
that are outside of the Timestamp range, for example.
Change-Id: Ib8ae7ebcbdb32e0aa58446bd1ef949e5e2f63e74
Related-Change: I23666ec8a067d829eaf9bfe54bd086c320b3429e
Related-Bug: 1821204
Partial-Bug: 1860149

Change-Id: I98ebe15353d675ca00fee387003bf6572ac385e6

Previously, we took the native string filename attribute and put it
directly in the (WSGI string) PATH_INFO field. Now, we convert it to
a WSGI string first.
Change-Id: I30e3beb8707b88c36bd3cdc7a0887d069e943ba6
Closes-Bug: #1858259 

unittest2 was needed for Python version <= 2.6, so it hasn't been needed
for quite some time. See unittest2 note one:
https://docs.python.org/2.7/library/unittest.html
This drops unittest2 in favor of the standard unittest module.
Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

Apparently FakeSwift has always been persnickety about the order of
query params of registered responses and the app making the calls.
Since query params can often be converted to dictionaries the order of
the encoded params should be able to change without effecting the apps
dispatching of registered responses.
Change-Id: Ied68c9334201a7663e9c85f3bdaa5b0643d4b6db