openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Make all statements using all caps (WARNING) and modify with print where there is stderr.write.
Fixes bug 1511139
Change-Id: I3b24c55d6ea07d9c2254f15a490fa336ad64c6c4

Tempauth fernet tokens use a secret shared among all proxies to encrypt
user group information. Because they are encrypted, clients can neither
view nor edit this information; it is an opaque bearer token similar to
the existing memcached-backed tokens (just much longer). Note that
tokens still expire after the configured token_life.
Add a new set of config options of the form
 fernet_key_<keyid> = <32 url-safe base64-encoded bytes>
Any of the configured keys will be used to attempt to decrypt tokens
starting with "ftk" and extract group information.
Another new config option
 active_fernet_key_id = <keyid>
dictates which key should be used when minting tokens. Such tokens
will start with "ftk" to distinguish them from memcached-backed tokens
(which continue to start with "tk"). If active_fernet_key_id is not
configured, memcached-backed tokens continue to be used.
Together, these allow seamless transitions from memcached-backed tokens
to fernet tokens, as well as transitions from one fernet key to another:
 1. Add a new fernet_key_<keyid> entry.
 2. Ensure all proxies have the new config with fernet_key_<keyid>.
 3. Set active_fernet_key_id = <keyid>.
 4. Ensure all proxies have the new config with the new
 active_fernet_key_id.
This is similar to the key-rotation process for the encryption feature,
except that old keys may be pruned following a token_life period.
Additionally, opportunistically compress groups before minting tokens.
Compressed tokens will begin with "zftk" but otherwise behave just like
"ftk" tokens.
Change-Id: I0bdc98765d05e91f872ef39d4722f91711a5641f

... so that eventlet.wsgi will include Connection: close headers on
Expect: 100-continue error responses from s3api and make aws clients
less unhappy.
eventlet.wsgi likes to check for hasattr('__len__', resp) when setting
connection close on expect-100 errors, but uses 'content-length' in
headers when deciding on chunked-transfer.
When we know the length we can support either interface. Also we can
imporove s3api to return error responses with the content-length known.
Change-Id: Ic504841714bd607cb9733b2de5126092a79c1094

Change-Id: I60f0b36633c2a348933fd45d348d76b256fca57a

Previously the SigV4Mixin would override S3Request signature checking
methods. This patch refactors the signature checking into sigv2 and
sigv4 helper classes i.e. moves towards composition rather than
inheritance.
No behavioural changes are intended with this patch.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thibault Person <thibault.person@ovhcloud.com>
Change-Id: Icaf86181c3bf7804c5176db48d2de5e2fc6f24d2

The method is redundant and confusingly named. Since the
Related-Change it actually returns the aggregate value of counters,
not just the count of calls made to the increment method alone.
Related-Change: I4d8a4b530465b587caced4362eb9178507701cef
Change-Id: I52585567a177f58e0f110785b44f8d238b2ac54d

Both the StatsdClient and LabeledStatsdClient implemented all of the
public interface methods for an AbstractStatsdClient. Even though the
implementations are identical the interfaces themselves were made
different WRT the "last" positional arg if provided.
This change puts the identical implementations of the public interface
in the AbstractStatsdClient class with a generic interface. For
backwards compat with the existing public interface on the
legacy-StatsdClient we have to make the old signature explicit - but we
can transparently call into the common implementation.
For consistency with legacy-Statsdclient and better docstrings this
change also "fixes" the signature for the new LabeledStatsdClient public
interface methods so they explicitly state labels must always be passed
in as a kwarg to avoid confusion accidentally passing in
`sample_rate=<dict>` or `label=<float>` as a positional arg.
Drive-by: as a side-effect of consolidating the implementations there
was now only one place to extend a fix in the FakeStatedClient interface
so that tests who assert StatsdClient method calls don't have to read
like the UUT also called some *other* public StatsdClient method; which
also cleaned up some Fake helpers that deal specifically with counters.
Drive-by: fix stub return value TypeError in a _get_hashes mock that was
caught by better faking. N.B. it doesn't matter how statsd would have
handled 'metric:None|c' because it was only a test bug.
Change-Id: I4d8a4b530465b587caced4362eb9178507701cef

Add test coverage for passing sample_rate to LabeledStatsdClient
methods.
Drive-by: clean up and tighten some existing StatsdClient unit tests to
share a CommonBaseTestsMixin, as was expedient perhaps to the chagrin of
the co-author.
Related-Change: I115ffb1dc601652a979895d7944e011b951a91c1
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I68a24d9467e58994abbac82a5e741f5bb578976c

Hopefully if we never do a release that supports signalfx no one will
ever use it and we won't have to maintain it.
Drive-by: refactor label model dispatch to fix a weird bug where a
config name could be a class attribute and blow up weird.
Change-Id: I2c67b59820c5ca094077bf47628426f4b0445ba0

Introduce a LabeledStatsdClient API; no callers yet.
Include three config options:
 - statsd_label_mode, which specifies which label format to use
 - statsd_emit_legacy, which dictates whether to emit old-style
 metrics dotted metrics
 - statsd_user_label_<name> = <value>, which supports user defined
 labels in restricted ASCII characters
Co-Authored-By: yanxiao@nvidia.com
Co-Authored-By: alistairncoles@gmail.com
Change-Id: I115ffb1dc601652a979895d7944e011b951a91c1

Change-Id: I5a0fa398901895de30ce979e22ef65fbd299656f

In test assertion failure messages the DUMMY_VALUE would be shown as
<object object ...> which is not immediately recognisable as the dummy
value..
This change makes the DUMMY_VALUE show as
<test.unit.common.middleware.helpers.FakeSwiftDummyValue object ...>.
Change-Id: I73fac245c65c48d45e4919c6f15688d482148407
Related-Change: I332ce724aa10287800cbec8ca21aacc3bbd3c22a

None values in the request environ should be copied, not replaced with
the dummy value.
Related-Change: I332ce724aa10287800cbec8ca21aacc3bbd3c22a
Change-Id: Id78ac1cac833f9284946cd364b3f95341afcb235

Change-Id: I9044e49ff30b092a97b75fa7fa133016d17cde37

Change-Id: Ibd3d5f83fcd31d78447a894ff4426e3e1b62e5a8

With py311 we started seeing warnings like
 DeprecationWarning: 'cgi' is deprecated and slated for removal
 in Python 3.13
The recommended replacement isn't quite up to snuff (doesn't handle
multiple parameters), but we already have a reasonably close replacement.
Add a "loose" mode to parse_content_type to make it suitable when there
may be a slash in a parameter token. Add a new utils.parse_header function
that takes advantage of the parse_content_type "loose" mode.
Closes-Bug: #2084472
Change-Id: Ie281ff90796f2d68840952c95669f264480b1b4c

We need to parse additional headers earlier, before stdlib tries to
establish message framing.
Now, TestReconstructorRebuildUTF8 can pass on py3.
Closes-Bug: #2097030
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Related-Change: https://review.opendev.org/c/openstack/swift/+/662546
Change-Id: I6aa16fda9285c9fc3816da6fbff2615bd14a020c

Change-Id: Iae07ec8d38cd747f8bfffa3c08b423bee4b6623c

There's a few places where bespoke file-like wrapper classes have been
implemented. The common methods are now inherited from
utils.InputProxy.
Make utils.FileLikeIter tolerate size=None to mean the same as size=-1
so that it is consistent with the behavior of other input streams.
Fix docstrings in FileLikeIter.
Depends-On: https://review.opendev.org/c/openstack/requirements/+/942845
Change-Id: I20741ab58b0933390dc4679c3e6b2d888857d577

It's always called with sha256 and raises sha256-specific errors;
we don't need to pretend to support arbitrary hashes.
Change-Id: Icff79ded067084249080e3e6f555429261eb0af0

...at least, provided the client sent a X-Amz-Content-SHA256 header.
Apparently Content-MD5 is no longer strictly required by AWS? Or maybe
it never was, provided the client sent a SHA256 of the content.
This also allows us to test with newer boto3, botocore, s3transfer.
Related-Bug: #2098529
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ifbcde9820bee72d80cab0fe3e67ea0f5817df949

Also:
- Couple small clean-ups in account.backend and account.utils
- Make a couple test assertions more useful.
Related-Change: https://review.opendev.org/c/openstack/swift/+/940601
Change-Id: Ic14642df50592c982adfb55a0c6cfd673a5a95b8

This implementation uses abstract sockets for process notifications,
similar to systemd's notify sockets, but notifiers use a PID-specific
name from a well-known namespace and listeners are assumed to be
ephemeral.
Update swift-reload to use these instead of polling child processes to
determine when a server reload has completed. Bonus: it also acts as a
non-blocking lock to prevent two swift-reload commands from reloading a
process at the same time.
Closes-Bug: #2098405
Related-Change: Ib2dd9513d3bb7c7686e6fa35485317bbad915876
Change-Id: I5f36aba583650bddddff5e55ac557302d023ea1b

Capture a copy of the request environ as part of FakeSwiftCall and use
that to query captured txn_ids, sources rather than maintaining
separate lists for each request property.
Store captured request body as an attribute of FakeSwiftCall to
similarly avoid maintaining another list. The request body is read
after the FakeSwiftCall has been captured in case there is an error
while reading.
Drive-by: drop footers arg from FakeSwiftCall constructor since it is
never used, nor should it be used.
Related-Change: If24b6fa50f1d67a7bbbf9a1794c70d37c41971f7
Change-Id: I332ce724aa10287800cbec8ca21aacc3bbd3c22a

Closes-Bug: #1674543
Change-Id: Ic74dbcaf6d8293ae41984d5cd61f0326c91988e2

Make encrypter unit test assertions more explicit by using assertions
on the named attributes of FakeSwiftCall rather than assertions on
position-indexed call tuples.
Change-Id: I871ddcc4ba559e7e4c0d0e28464780c6cd669797

The encrypter middleware uses an update_footers callback to send
request footers. Previously, FakeSwift combined footers with captured
request headers in a single dict. Tests could not therefore
specifically assert that *footers* had been captured rather than
headers.
This patch modifies FakeSwift to capture footers separately for each
request. Footers are still merged with the request headers in order to
synthesise GET or HEAD response headers when a previously uploaded
object is returned.
Unfortunately the change cannot be as simple as adding another
attribute to the FakeSwiftCall namedtuple. A list of these namedtuples
is returned by FakeSwift.calls_with_headers. Some tests cast the
namedtuples to 3-tuples and will break if the length of the namedtuple
changes. Other tests access the attributes of the namedtuples by name
and will break if the list values are changed to plain 3-tuples.
Some test churn is therefore inevitable:
* FakeSwiftCall is changed from a namedtuple to a class. This prevents
 future tests assuming it is a fixed length tuple. It also supports a
 headers_and_footers property to return the combination of uploaded
 headers and footer that was previously (confusingly) returned by
 FakeSwiftCall.headers.
* A new property FakeSwift.call_list has been added which returns a
 list of FakeSwiftCalls.
* FakeSwift.calls_with_headers now returns a 3-tuple. Tests that
 previously assumed this was a namedtuple have been changed to use
 FakeSwift.call_list instead, which gives them objects with the same
 named attributes as the previous namedtuple. Tests that previously
 treated the namedtuple as a 3-tuple do not need to be changed.
* Tests that access the 'private' FakeSwift._calls have been changed
 to use FakeSwift.call_list.
Change-Id: If24b6fa50f1d67a7bbbf9a1794c70d37c41971f7

The following configuration options are deprecated:
 * expiring_objects_container_divisor
 * expiring_objects_account_name
The upstream maintainers are not aware of any clusters where these have
been configured to non-default values.
UpgradeImpact:
Operators are encouraged to remove their "container_divisor" setting and
use the default value of 86400.
If a cluster was deployed with a non-standard "account_name", operators
should remove the option from all configs so they are using a supported
configuration going forward, but will need to deploy stand-alone expirer
processes with legacy expirer config to clean-up old expiration tasks
from the previously configured account name.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Jianjian Huo <jhuo@nvidia.com>
Change-Id: I5ea9e6dc8b44c8c5f55837debe24dd76be7d6248

Add the storage_policy attribute to the metadata returned
when listing containers using the GET account API function.
The storage policy of a container is a very useful attribute
for telemetry and billing purposes, as it determines the location
and method/redundancy of on-disk storage for the objects in the
container. Ceilometer currently cannot define the storage policy as a
metadata attribute in Gnocchi as GET account, the most efficient way
of discovering all containers in an account, does not return the
storage policy for each container.
Returning the storage policy for each container in GET account
is the ideal way of resolving this issue, as it allows Ceilometer
to find all containers' storage policies without performing additional
costly API calls.
Special care has been taken to ensure the change is backwards
compatible when migrating from pre-storage policy versions
of Swift, even though those versions are quite old now.
This special handling can be removed if support for migrating
from older versions is discontinued.
Closes-bug: #2097074
Change-Id: I52b37cfa49cac8675f5087bcbcfe18db0b46d887

Previously, test_object_move_no_such_object_no_tombstone_ancient
would fail intermittently, with an assertion that two timestamps
were almost (but not quite) equal.
This probably comes down to the fact that it's passing floats as
timestamps down into FakeInternalClient's parse(); specifically,
values like 1738046018.2900746 and 1738045066.1442454 are known
to previously fail.
Just fixing the usage doesn't fix the foot-gun, though -- so fix
up parse() to be internally consistent, even if passed a float.
Change-Id: Ide1271dc4ef54b64d2dc99ef658e8340abb0b6ce