openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: Id123fb0f9173cf75916984d85bcf8c2d5eefaec2

Change-Id: Ic07b0bdd5f1c90e7e25991def58c587680d7bfab

As per victoria cycle testing runtime and community goal[1]
we need to migrate upstream CI/CD to Ubuntu Focal(20.04).
Moving py38 job to focal.
Story: #2007865
Task: #40221
[1] https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal.h>
Change-Id: I2a1ce0bda37a4764f7e56a9c15a283eefa46e17b

This patch changes 'Seperate' to
'Separate' in the 361 line of
swift/common/request_helpers.py
Change-Id: I438e9503818c12440e1534938fdba857350007b6

A new header `X-Backend-Use-Replication-Network` is added; if true, use
the replication network instead of the client-data-path network.
Several background daemons are updated to use the replication network:
 * account-reaper
 * container-reconciler
 * container-sharder
 * container-sync
 * object-expirer
Note that if container-sync is being used to sync data within the same
cluster, the replication network will only be used when communicating
with the "source" container; the "destination" traffic will continue to
use the configured realm endpoint.
The direct and internal client APIs still default to using the
client-data-path network; this maintains backwards compatibility for
external tools written against them.
UpgradeImpact
=============
Until recently, servers configured with
 replication_server = true
would only handle REPLICATE (and, in the case of object servers, SSYNC)
requests, and would respond 405 Method Not Allowed to other requests.
When upgrading from Swift 2.25.0 or earlier, remove the config option
and restart services prior to upgrade to avoid a flood of background
daemon errors in logs.
Note that some background daemons find work by querying Swift rather
than walking local drives that should be available on the replication
network:
 * container-reconciler
 * object-expirer
Previosuly these may have been configured without access to the
replication network; ensure they have access before upgrading.
Closes-Bug: #1883302
Related-Bug: #1446873
Related-Change: Ica2b41a52d11cb10c94fa8ad780a201318c4fc87
Change-Id: Ieef534bf5d5fb53602e875b51c15ef565882fbff

Previously, we would work with these paths as WSGI strings -- this would
work fine when all data were read and written on the same major version
of Python, but fail pretty badly during and after upgrading Python.
In particular, if a py3 proxy-server tried to read existing data that
was written down by a py2 proxy-server, it would hit an error and
respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
was freshly-written by a py3 proxy, it would serve corrupt data back to
the client (including a corrupt/invalid ETag and Content-Type).
Now, ensure that both py2 and py3 write down paths as native strings.
Make an effort to still work with WSGI-string metadata, though it can be
ambiguous as to whether a string is a WSGI string or not. The heuristic
used is if
 * the path from metadata does not match the (native-string) request
 path and
 * the path from metadata (when interpreted as a WSGI string) can be
 "un-wsgi-fied" without any encode/decode errors and
 * the native-string path from metadata *does* match the native-string
 request path
then trust the path from the request. By contrast, we usually prefer the
path from metadata in case there was a pipeline misconfiguration (see
related bug).
Add the ability to read and write a new, unambiguous version of metadata
that always has the path as a native string. To support rolling
upgrades, a new config option is added: meta_version_to_write. This
defaults to 2 to support rolling upgrades without configuration changes,
but the default may change to 3 in a future release.
UpgradeImpact
=============
When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set
 meta_version_to_write = 1
in your keymaster's configuration. Regardless of prior Swift version, set
 meta_version_to_write = 3
after upgrading all proxy servers.
When switching from Python 2 to Python 3, first upgrade Swift while on
Python 2, then upgrade to Python 3.
Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
Closes-Bug: #1888037
Related-Bug: #1813725 

When I call the S3 API using the AWS .NET SDK, I get the following error.
An error occurred (AuthorizationHeaderMalformed) when calling the
ListBuckets operation: The authorization header is malformed;
the region 'regionone' is wrong; expecting 'RegionOne'
The reason is that the AWS .NET SDK generates a signature by changing
the region name to lowercase. (AWS region names are all lowercase.)
The default region name of OpenStack is RegionOne, and custom region
names with capital letters can also be used.
If you set the location of the S3 API to a region name containing
uppercase letters, the AWS .NET SDK cannot be used.
There are two ways to solve this problem.
1. Force the location item of S3 API middleware setting to be set
 to lower case.
2. If the request contains credentail parameters that contain the
 lowercase region name, the region name of string_to_sign is
 modified to lowercase to generate a valid signature.
I think the second way is to make it more compatible.
Closes-Bug: #1888444
Change-Id: Ifb58b854b93725ed2a1e3bbd87f447f2ab40ea91

It does *not* include transfer time; it's just the time until response
is started.
Change-Id: I67c5c37c8c669e1e2de2cbfca8bcec02dc98659f

We weren't actually willing to change anything; sure sounds like an
error to me.
Change-Id: I033f6fba4ddb3dd502dfd5393dd2e9a2115672e0

When you have overlapping active shard ranges updates will get sent to
"the first" database; but when the proxy queries shards for listings
they get stitched together end-to-end with markers.
This means mostly the second shard range is ignored. But since the
order of shard ranges is not stable (it seems to depend on the database
indexes; which can change when rows are added or removed) you could send
updates to "the wrong" shard.
Using a stable order leads to more correct and robust behavior under
failure; and is also better for cognitive overhead.
Change-Id: Ia9d29822bf07757fc1cf58ded90b49f12b7b2c24

Previously, the replication_server setting could take one of three
states:
 * If unspecified, the server would handle all available methods.
 * If "true", "yes", "on", etc. it would only handle replication
 methods (REPLICATE, SSYNC).
 * If any other value (including blank), it would only handle
 non-replication methods.
However, because SSYNC tunnels PUTs, POSTs, and DELETEs through
the same object-server app that's responding to SSYNC, setting
`replication_server = true` would break the protocol. This has
been the case ever since ssync was introduced.
Now, get rid of that second state -- operators can still set
`replication_server = false` as a principle-of-least-privilege guard
to ensure proxy-servers can't make replication requests, but replication
servers will be able to serve all traffic. This will allow replication
servers to be used as general internal-to-the-cluster endpoints, leaving
non-replication servers to handle client-driven traffic.
Closes-Bug: #1446873
Change-Id: Ica2b41a52d11cb10c94fa8ad780a201318c4fc87

Also, drop translation for the error message to improve search-ability.
Change-Id: I24e417ca7500bc9455d5307e6712517c75b2cd25
Partial-Bug: 1674543

The current behavior is really painful when you've got hundreds of shard
ranges in a DB. The new summary with the states is default. Users can
add a -v/--verbose flag to see the old full detail view.
Change-Id: I0a7d65f64540f99514c52a70f9157ef060a8a892

Tests that trigger error limiting can bleed state across the long lived
module servers. Expand usage of existing pattern to clear error
limiting in TestCase setUp and tearDown.
Change-Id: I0dac997f9ff2c7c60e9769ef6d4d802ea7d71ca8

We've seen failures with probe tests lately where dnspython 2.0.0 is
getting installed even though it doesn't support py2 anymore. I think
using latest pip should be better about noticing that and installing the
last 1.x release intead?
Change-Id: I6eda54ccd2792effadb334ce9324887132b62b6f

Note that existing SAIOs with 60xx ports should still work fine.
Change-Id: If5dd79f926fa51a58b3a732b212b484a7e9f00db
Related-Change: Ie1c778b159792c8e259e2a54cb86051686ac9d18

Lower the part-power -- 18 is way higher than is needed for a dev
environment.
Add commands for reduced-redundancy and erasure-coded storage policies.
Related-Change: Ibe46011d8e6a6482d39b3a20ac9c091d9fbc6ef7
Related-Change: I6f11f7a1bdaa6f3defb3baa56a820050e5f727f1
Related-Change: I0403016a4bb7dad9535891632753b0e5e9d402eb
Change-Id: I13de27674c81977c2470d43bbb2126ecc4bdd85a

It's not actually *used* anywhere.
Change-Id: I8f9b5cf7f5749481ef391a2029b0c4263443a89b

Change-Id: I918ab4ccbf4d081b26f4117937410cdad1caf8d3
Closes-Bug: #1862645
Closes-Bug: #1886782 

My customers use the upstream documentation for manual
installation of Swift nodes and they come upon the lack
of any mention that host firewall ought to accomodate
services running on the node. Perhaps we thought it
self-evident in Swift.
Related RH bz#: 1797814
Change-Id: I337f8d0f1fbeee7ae927a581eecbbbcc6dc69340

In the previous patch, we could clean up all container DBs, but only if
the daemons went in a specific order (which cannot be guaranteed in a
production system).
Once a reclaim age passes, there's a race: If the container-replicator
processes the root container before the container-sharder processes the
shards, the deleted shards would get reaped from the root so they won't
be available for the sharder. The shard containers then hang around
indefinitely.
Now, be willing to mark shard DBs as deleted even when we can't find our
own shard range in the root. Fortunately, the shard already knows that
its range has been deleted; we don't need to get that info from the root.
Change-Id: If08bccf753490157f27c95b4038f3dd33d3d7f8c
Related-Change: Icba98f1c9e17e8ade3f0e1b9a23360cf5ab8c86b

When a DB gets deleted, we clear out its metadata. This included sysmeta
such as that used to tell shards the name of their root DB.
Previously, this would cause deleted shards to pop back to life as roots
that claimed to have objects still sitting in whatever container they
sharnk into.
Now, use the metadata if it's available, but when it's not, go by the
state of the DB's "own shard range" -- deleted shards should be marked
deleted, while roots never are.
This allows us to actually clean up the database files; you can test
this by doing something like
 * Run `nosetests test/probe/test_sharder.py:TestContainerSharding.test_shrinking`
 * Run `find /srv/*/*/containers -name '*.db'` to see how many databases
 are left on disk. There should be 15: 3 for the root container, 6 for
 the two shards on the first pass, and another 6 for the two shards on
 the second pass.
 * Edit container configs to decrease reclaim_age -- even 1 should be
 fine.
 * Run `swift-init main start` to restart the servers.
 * Run `swift-init container-sharder once` to have the shards get marked
 deleted.
 * Run `swift-init container-updater once` to ensure all containers have
 reported.
 * Run `swift-init container-replicator once` to clean up the
 containers.
 * Run `find /srv/*/*/containers -name '*.db'` again to verify no
 containers remain on disk.
Change-Id: Icba98f1c9e17e8ade3f0e1b9a23360cf5ab8c86b

... and to determine {account}, {container}, and {object} template
values, as well as statsd metric names.
UpgradeImpact:
--------------
Be aware that this will cause an increase in the proxy-logging statsd
metrics emited for s3api responses. However, this will more accurately
reflect the state of the system.
Change-Id: Idbea6fadefb2061f83eed735ef198b88ba7aaf69

In a running proxy server, this only ever comes from one place:
a key in the wsgi env, populated by the memcache middleware. By
littering proxy controller code with things like
 memcache = getattr(app, 'memcache', None) or env.get('swift.cache')
we cause ourselves to second-guess that.
Change-Id: Ia652b1381990b61194978c4efaebac7d2a602ca3

We rely on having byte-accurate representations of the request path
as-seen-on-the-wire to compute signatures in s3api; the unquoting /
requoting madness to make non-ascii paths work with python's stdlib can
lead to erroneous SignatureDoesNotMatch responses.
Change-Id: I87fe3477d8b7ef186421ef2d08bc3b205c18a0c1
Closes-Bug: #1884991 

Change-Id: If0fc8ec4d8056afb741bf74b82598a26683dfcd7

If an object name has something like /./ in it then curl will
resolve this. Need to use --path-as-is option for curl
Change-Id: I4e45cb62d41f6aada4fdbb00d86b4bd737b441ee
Closes-Bug: #1885244 

When completing a multipart-upload, include the upload-id in sysmeta.
If we can't find the upload marker, check the final object name; if it
has an upload-id in sysmeta and it matches the upload-id that we're
trying to complete, allow the complete to continue.
Also add an early return if the already-completed upload's ETag matches
the computed ETag for the user's request. This should help clients that
can't take advantage of how we dribble out whitespace to try to keep the
conneciton alive: The client times out, retries, and if the upload
actually completed, it gets a fast 200 response.
Change-Id: I38958839be5b250c9d268ec7c50a56cdb56c2fa2