openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I05336b700120ab5fcf922590d6a12f73112edb50

The timing stats decorators are moved from utils to base_storage_server.py
to avoid a circular import of HTTPException.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Idc4b52de0a04ebfc0e353162bd791d4e0e20eac3

openSUSE stopped providing OpenStack packages some time ago.
Co-authored-by: Takashi Kajinami <kajinamit@oss.nttdata.com>
Change-Id: I60fcbac1cb6ec370de565903375be411e8fc3415

pip 23.1 removed the "setup.py install" fallback for projects
that do not have pyproject.toml and now uses a pyproject.toml
which is vendored in pip.
To address that, this change adds the minimal pyproject.toml
to enable pbr to be properly used to build editable wheels.
This is required to support installing devstack on
centos stream 9 and related distros with GLOBAL_VENV=True
Without this change the wsgi scripts are not generated in
editable mode. i.e. pip install -e /opt/stack/keystone
See https://pip.pypa.io/en/stable/news/#v23-1
and https://github.com/pypa/pip/issues/8368 for more
details on the removal of the fallback support.
setuptools v64.0.0 is used to support editable installs
via its PEP-660 implmentation
https://github.com/pypa/setuptools/pull/3488
Change-Id: I82e0c130c4022164bc8e4d1120aae9bdb9a43b5d

This got missed when we dropped support for py36.
Related-Change: I6f0cbde23f0a0db75bdb3e598ee862af0bad0dba
Change-Id: I3dcca36744bbdbce7e5e22e50b3a3fa263071717

We recently removed the "func-py3" tox env in favor of just "func" -- as a
result, this went back to running unit tests!
Keep "func-py3" for older releases where that tox env *is* still defined,
though, as they won't have py27 around to run "func".
Related-Change: I53153c4fde043e964e1daa7bbf2089e0471dede2
Change-Id: Ibbb8c305a8b8bac7ca1a8853208cd031957adac1

Some tests previously made assertions on statsd client methods that
were implemented by a delegated interface patched onto a logger
instance. Those tests have run against a StatsdClient since the
Related Change, so the use of the term 'delegate' in tests no longer
makes sense.
Change-Id: Ie47fa50a226da6ff7b067d6a367758ac9f12ef4c
Related-Change: I3a677bb67c5700da48f89c847f652b4610ba47c2

Change-Id: I6f0cbde23f0a0db75bdb3e598ee862af0bad0dba

A new CRCHasher class wraps crc functions to mimic the hashlib
hasher interface.
For crc32, use the zlib implementation.
For crc32c, use ISA-L if we can find it; on py38+ this includes the
ISA-L bundled in pyeclib's binary wheels.
If ISA-L is not available, check for kernel crypto API support for crc32c;
use it if available. Otherwise, raise a NotImplementedError.
No callers yet, but these will be used by s3api in a later patch.
Change-Id: Ic0c55e307ce10b56b569c9fee728c445a2300cbd

This update implements Sigv4-streaming (chunked upload) as described in
the Amazon S3 documentation:
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
Closes-Bug: #1810026
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: ashnair <ashnair@nvidia.com>
Change-Id: I7be1ce9eb5dba7b17bdf3e53b0d05d25ac0a05b0

The file is not used because PBR is used.
Change-Id: Iefef64f63cbd8847a6b0322edc4a8ce85a6b8580

This will prevent client timeouts during large object copies, similar to existing support in
bulk operations and SLO manifest writes. The operator can set the frequency using the
yield_frequency inside conf file.
To use heartbeat, simply add "heartbeat=on" query parameter, which will:
- Returns a 202 Accepted response
- Sends periodic 1 byte whitespace heartbeat every yield_frequency
- Returns final status and headers
- Supports both plain text and JSON
Closes-Bug: #1738306
Change-Id: Iaf4355f2e0edbd6dde226231f3d7a61a5e53ecfa

Bobcat went EOL recently; add Epoxy job.
Related-Change: https://review.opendev.org/c/openstack/releases/+/948240
Change-Id: Ibad603c4a07d49ce55dce26d1af4c506d95b193f

Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Change-Id: I155572bc5f4d955c918c951d6d2af9dbbae0136d

Make all statements using all caps (WARNING) and modify with print where there is stderr.write.
Fixes bug 1511139
Change-Id: I3b24c55d6ea07d9c2254f15a490fa336ad64c6c4

The tags framework has been discontinued for a long time; see
https://governance.openstack.org/tc/reference/tags/ and
https://governance.openstack.org/tc/resolutions/20211224-tags-framework-removal.html
Related-Change: I78815177dba9d765587d9a0aee4e0f76e14a043e
Change-Id: Ifec53e00b760a7c3ee82b7ef3d54b8f856c55aba

Tempauth fernet tokens use a secret shared among all proxies to encrypt
user group information. Because they are encrypted, clients can neither
view nor edit this information; it is an opaque bearer token similar to
the existing memcached-backed tokens (just much longer). Note that
tokens still expire after the configured token_life.
Add a new set of config options of the form
 fernet_key_<keyid> = <32 url-safe base64-encoded bytes>
Any of the configured keys will be used to attempt to decrypt tokens
starting with "ftk" and extract group information.
Another new config option
 active_fernet_key_id = <keyid>
dictates which key should be used when minting tokens. Such tokens
will start with "ftk" to distinguish them from memcached-backed tokens
(which continue to start with "tk"). If active_fernet_key_id is not
configured, memcached-backed tokens continue to be used.
Together, these allow seamless transitions from memcached-backed tokens
to fernet tokens, as well as transitions from one fernet key to another:
 1. Add a new fernet_key_<keyid> entry.
 2. Ensure all proxies have the new config with fernet_key_<keyid>.
 3. Set active_fernet_key_id = <keyid>.
 4. Ensure all proxies have the new config with the new
 active_fernet_key_id.
This is similar to the key-rotation process for the encryption feature,
except that old keys may be pruned following a token_life period.
Additionally, opportunistically compress groups before minting tokens.
Compressed tokens will begin with "zftk" but otherwise behave just like
"ftk" tokens.
Change-Id: I0bdc98765d05e91f872ef39d4722f91711a5641f

Change-Id: Ib797f73134c8fa5ef0dc66aeddf06adfa50ed7d4

Now that we no longer need to support py2, iterating over bytes always
yields ints.
Change-Id: Id5e0c4f597c77cae5742c53f31fad479fa864261

... so that eventlet.wsgi will include Connection: close headers on
Expect: 100-continue error responses from s3api and make aws clients
less unhappy.
eventlet.wsgi likes to check for hasattr('__len__', resp) when setting
connection close on expect-100 errors, but uses 'content-length' in
headers when deciding on chunked-transfer.
When we know the length we can support either interface. Also we can
imporove s3api to return error responses with the content-length known.
Change-Id: Ic504841714bd607cb9733b2de5126092a79c1094

Change-Id: I60f0b36633c2a348933fd45d348d76b256fca57a

Previously the SigV4Mixin would override S3Request signature checking
methods. This patch refactors the signature checking into sigv2 and
sigv4 helper classes i.e. moves towards composition rather than
inheritance.
No behavioural changes are intended with this patch.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thibault Person <thibault.person@ovhcloud.com>
Change-Id: Icaf86181c3bf7804c5176db48d2de5e2fc6f24d2

Introduce a single error handler to catch errors while reading the
input.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ib0330e863ad15e362dfa587517a26109c74eefec

Currently our test names are long and we pipe pytest output to a file,
so the skip reason gets truncated; setting this env var .zuul and
passing to tox.ini will allow us to see the reason why pytest is
skipping next to the test name.
Drive-by: remove passenv from s3api tests in tox.ini so they are
inherited from testenv
Related-Change-Id: I6b7cc64262d2f8ef7e3bbce30bab5550899b5211
Change-Id: I15791a9238485190acf7f87a3b562bf67e7c5c09

The swift-tox-func-s3api-tests-tempauth runs the tests in test/s3api.
The swift-tox-func-s3api-ceph-s3tests-tempauth doesn't run the tests
in test/s3api. This patch changes the names of those jobs to
disambiguate the prefix 'swift-tox-func-s3api' and avoid any
suggestion that the second job is a super-set of the first job.
Change-Id: I323cf7e32200a2f895f0b051454b00047f7de300

The method is redundant and confusingly named. Since the
Related-Change it actually returns the aggregate value of counters,
not just the count of calls made to the increment method alone.
Related-Change: I4d8a4b530465b587caced4362eb9178507701cef
Change-Id: I52585567a177f58e0f110785b44f8d238b2ac54d

Both the StatsdClient and LabeledStatsdClient implemented all of the
public interface methods for an AbstractStatsdClient. Even though the
implementations are identical the interfaces themselves were made
different WRT the "last" positional arg if provided.
This change puts the identical implementations of the public interface
in the AbstractStatsdClient class with a generic interface. For
backwards compat with the existing public interface on the
legacy-StatsdClient we have to make the old signature explicit - but we
can transparently call into the common implementation.
For consistency with legacy-Statsdclient and better docstrings this
change also "fixes" the signature for the new LabeledStatsdClient public
interface methods so they explicitly state labels must always be passed
in as a kwarg to avoid confusion accidentally passing in
`sample_rate=<dict>` or `label=<float>` as a positional arg.
Drive-by: as a side-effect of consolidating the implementations there
was now only one place to extend a fix in the FakeStatedClient interface
so that tests who assert StatsdClient method calls don't have to read
like the UUT also called some *other* public StatsdClient method; which
also cleaned up some Fake helpers that deal specifically with counters.
Drive-by: fix stub return value TypeError in a _get_hashes mock that was
caught by better faking. N.B. it doesn't matter how statsd would have
handled 'metric:None|c' because it was only a test bug.
Change-Id: I4d8a4b530465b587caced4362eb9178507701cef

Change-Id: I6b7cc64262d2f8ef7e3bbce30bab5550899b5211

Add test coverage for passing sample_rate to LabeledStatsdClient
methods.
Drive-by: clean up and tighten some existing StatsdClient unit tests to
share a CommonBaseTestsMixin, as was expedient perhaps to the chagrin of
the co-author.
Related-Change: I115ffb1dc601652a979895d7944e011b951a91c1
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I68a24d9467e58994abbac82a5e741f5bb578976c