openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Newer versions of python expect to be able to say `with self.lock:`
down in logging.
See https://github.com/python/cpython/commit/74723e11
Change-Id: I30305566d12a1b8be4c8bde4b416b798322a1385

It is currently possible to configure "X-Container-Meta-Quota-Bytes"
and "X-Container-Meta-Quota-Count" on containers, as well as
"X-Account-Meta-Quota-Bytes" on accounts. However, there is no way
to configure an account quota limit for the number of files or
"quota-count". This limitation could potentially allow a user to
exhaust filesystem inodes.
This change introduces the "X-Account-Meta-Quota-Count" header,
allowing users with the ResellerAdmin role to add a quota-count
limit on accounts. The middleware will enforce this limit similarly
to the existing quota-byte limit.
Co-authored-by: Azmain Adib <adib1905@gmail.com>
Co-authored-by: Daanish Khan <daanish1337@gmail.com>
Co-authored-by: Mohamed Hassaneen <mohammedashoor89@gmail.com>
Co-authored-by: Nada El-Mestkawy <nadamaged05@gmail.com>
Co-authored-by: Tra Bui <trabui.0517@gmail.com>
Change-Id: I1d11b2974de8649a111f2462a8fef51039e35d7f

When the proxy passes the container-update headers to the object server
now include the db_state, which it already had in hand. This will be
written to async_pending and allow the object-updater to know more about
a container rather then just relying on container_path attribute.
This patch also cleans up the PUT, POST and DELETE _get_update_target
paths refactoring the call into _backend_requests, only used by these
methods, so it only happens once.
Change-Id: Ie665e5c656c7fb27b45ee7427fe4b07ad466e3e2

Change-Id: I022404cd90d9755a09c20619c3a72588d3367467

Previously, `twine check <swift wheel>` would report
 WARNING `long_description_content_type` missing.
 defaulting to `text/x-rst`.
Change-Id: I5d004218515d17b8cbace46b2c1e646f734779f3

The ssl.wrap_socket method has been removed in 3.12.
SSLContext.wrap_socket should now be used.
Change-Id: I6119e054289eac263ff5448d7d118209f98678d9

Continue also tagging it "py3" so any users of that tag don't become
stuck in time.
Closes-Bug: #2037268
Closes-Bug: #2070029
Change-Id: I38d9469238d2eb6647414c1107e68ff6f3a15797

Something about them threw off the exit-immediately-on-non-zero-exit
behavior when building.
Related-Bug: #2070029
Change-Id: I3e40ebd78a9f8e93c905b24a12f5f54b2d27c2d9

The proxy may use a sub-request to fetch a container listing response,
for example for updating shard ranges. If it then fails to parse the
response, it should use the sub-request path when logging a warning or
error. Before, the proxy would use the original request path which may
have been to an object.
Change-Id: Id904f4cc0f911f9e6e53d4b3ad7f98aacd2b335d

Change-Id: If15c3fee706d295de678f812839a5d0759089ccf

Use same assertion pattern as introduced in the Related-Change and
include the host and port in the assertion.
Change-Id: I7cccf3b8cdfe157c27fb8e0b1f432a649970a865
Related-Change: I9a016de1010dd17e91bc85877413b34d2c3287c7

With labeled metrics on the horizon, I wanted some more end-to-end
tests of stats emission.
Change-Id: I9a016de1010dd17e91bc85877413b34d2c3287c7

It takes a config dict and some caller-specific options, similar to
get_logger. Use this in get_logger, so our logging module doesn't
need to know anything about statsd config options.
Co-Authored-By: yanxiao@nvidia.com
Change-Id: I5ae2cc5c257fb8d7eab885977d9d9cf602224ec7

There's a variety of reasons why s3api may return a 4xx or 5xx
response. It's useful to get some more detail in the log message,
particularly when the request is a HEAD which won't get any detail in
the response body.
Change-Id: I0a499dc5c50bb534a23adcbfe7c3822d8954b0e0

Noticed a few more projects had final py2 releases.
Change-Id: I144efd5ee670a5a6b444fbc31e9434fac31bf2ec

For some reason, when we switched from py36 on centos8 to py39 on
centos9, these two tests started failing. Looks like a disagreement
about whether the canonical path for a bucket request should have
a trailing slash or not.
Mark them as known-failures for now so we can stay aware of any
other new breakage brought on by swift code changes.
Related-Change: I4f6b9c07af7bc768654f1a5d0c66b048e0f2c9c1
Change-Id: If990752c7ef7667182dbe18e49679e48c0e3d42d

The old [files]scripts method of specifying executable Python scripts
triggers some legacy easy-install-like mode for editable installs,
which relies on pkg_resources. Recent versions of setuptools (67.5.0+)
have started emitting warnings when importing pkg_resources, which in
turn cause quite noticeable slowdowns in process startup. This is
particularly prominant on py312, which stopped pre-installing (an often
older version of) setuptools in new venvs.
See also:
 - https://github.com/python/cpython/issues/95299
 - https://github.com/pypa/setuptools/pull/3843
 - https://github.com/pypa/setuptools/issues/3966
Now, use [entry_points]console_scripts to specify these executables,
which does not use pkg_resources in the generated script files.
Change-Id: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83

While switching how some executable scripts were configured, I saw
some strange rolling-upgrade failures that seemed to indicate that
the new invocation method was trying to be used with old code. It
seems like it maybe has something to do with whether swift was
installed to /usr/local/lib/python3.9/site-packages/ or
/usr/local/lib64/python3.9/site-packages/ but I'm not entirely
sure.
At any rate, a proper package manager ought to uninstall the old
version then install the new one, so it seems reasonable to do that
with pip, too.
Change-Id: I12e84745e7601d162755bc9d0f1cda7b63e92197

Under a multi-region deployment with a single Keystone server,
specifying the Keystone auth credentials isn't enough. Indeed,
Castellan succeeds when logging-in, but may use the wrong
Barbican endpoint (if there are 2 Barbican deployed). This is
what happened to us, when deploying our 2nd region.
They way to fix it would be to tell Castellan what region to use,
unfortunately, there's no such option in Castellan. Though we may
specify the barbican_endpoint, which is what this patch allows.
Change-Id: Ib7f4219ef5fdef65e9cfd5701e28b5288741783e

The size in bytes from object metadata of expiring objects are stored in
expirey queue entries under the content_type field.
The x-content-type-timestamp take from object metadata is provided along
with the x-content-type update so the container replicator resolves the
latest content-type and ensures eventual consistency.
UpgradeImpact: During rolling upgrades you should expect expirer queue
entries to continue lacking swift_expirer_bytes= annotations until ALL
object servers replicas have been upgraded to new code.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ie4b25f1bd16def4069878983049b83de06f68e54

Remove swift-tox-py36-centos-8-stream job entirely.
Move the following jobs to CentOS 9:
 - swift-tox-func-s3api-ceph-s3tests-tempauth
 - swift-tox-func-s3api-tests-tempauth
 - swift-multinode-rolling-upgrade, as well as the other rolling
 upgrade jobs
Remove the swift-multinode-rolling-upgrade-victoria job, as py39
support (required for CentOS 9) was not added until wallaby.
Change-Id: I4f6b9c07af7bc768654f1a5d0c66b048e0f2c9c1

There was an abandoned change that made reference to a RecussionError
when running a probe test that imported boto3 that had something to do
with eventlet, ssl and a transitive dependency on requests-mock, but the
fix that actually got merged seemed to depend on another change to
tox.ini that disables request-mock when we run pytest.
Either way, we already import from boto3 at the top of probe tests and
it's in test-requirements; so we require it to be installed even if you
don't have s3api in your pipeline.
Related-Change: I789b257635c031ac0cb6e4b5980f741e0cb5244d
Related-Change: I2793e335a08ad373c49cbbe6759d4e97cc420867
Related-Change: If14e4d2c1af2efcbc99e9b6fe10973a7eb94d589
Change-Id: Id2662bfc5ef2f21f901f1c98e6389c4cb01818a2

Change-Id: Ie5d8b555489d28c3b901e5bbebdcecbde7bb3367

Change-Id: I9413c72f41465fb8026848f71ec3b39fa990c3b7

Drive-by: fix ValueError message for non_negative_int
Change-Id: I06508279d59fa57296dd85548f271a7812aeb45f

We've been working toward separating our logger from our statsd client.
This is generally a good idea; it's always been a little weird to have
our special-case loggers that would allow you to *also* increment some
counters.
The end goal is to take a bunch of places that look like
 logger = utils.get_logger(conf)
 ...
 logger.info(...)
 logger.increment(...)
and turn them into something more like
 logger = logs.get_adapted_logger(conf)
 stats = statsd_client.get_statsd_client(conf, logger=logger)
 ...
 logger.info(...)
 stats.increment(...)
Take a lesson from logging: callers don't need to know whether the
log_level is high enough that their message will be logged, or even
whether logging is enabled at all. Code wanting to emit stats shouldn't
need to know whether statsd collection has been configured, either.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I6eb5b27a387cc2b7310ee11cc49d38fd2b6cbab8