openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Related-Change: Icff8cf57474dfad975a4f45bf2d500c2682c1129
Change-Id: Ic2283fab0d18ea03c6beb353c6b934344606c15e

Some tooling out there, like Ansible, will always call to see if
object-lock is enabled on a bucket/container. This fails as Swift doesn't
understand the object-lock or the get object lock api[0].
When you use the get-object-lock-configuration to a bucket in s3 that
doesn't have it applied it returns a specific 404:
 GET /?object-lock HTTP/1.1" 404 None
 ...
 <?xml version="1.0" encoding="UTF-8"?>
 <Error>
 <Code>ObjectLockConfigurationNotFoundError</Code>
 <Message>Object Lock configuration does not exist for this bucket</Message>
 <BucketName>bucket_name</BucketName>
 <RequestId>83VQBYP0SENV3VP4</RequestId>
 </Error>'
This patch doesn't add support for get_object lock, instead it always
returns a similar 404 as supplied by s3, so clients know it's not
enabled.
Also add a object-lock PUT 501 response.
[0] https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html
Change-Id: Icff8cf57474dfad975a4f45bf2d500c2682c1129

If swift is used with an Application Credential with access_rules [1],
the catalog needs to be sent with the token for it to be evaluated by
keystonemiddleware.
[1]: https://docs.openstack.org/keystone/2023.1/user/application_credentials.html#access-rules
Change-Id: I0b02821722ffc819bc8a97281a33391072242ffc

Change-Id: I6ef909e826d3901f24d3c42a78d2ab1e4e47bb64

Personally I'm not a big fan of how we arrange logs for SAIO,
but it is a historic standard. The reconciler has to conform.
Change-Id: I45a25ff406b31b6b1b403e213554aaabfebc6eb5

The never ending changes in package names march on.
This is for Fedora 38.
Change-Id: Idffd1df7e2f23681eb7d7bf1717471b594612dbc

Change-Id: I5e75a359660315a34ec36f5cb748808257f4185a

Currently, SLO manifest files will be evicted from page cache
after reading it, which cause hard drives very busy when user
requests a lot of parallel byte range GETs for a particular
SLO object.
This patch will add a new config 'keep_cache_slo_manifest', and
try keeping the manifest files in page cache by not evicting them
after reading if config settings allow so.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I557bd01643375d7ad68c3031430899b85908a54f

Change-Id: I6fc9eaf0903bdbe4ccb82c4e7647478a1eef9785

Offer it both by service and as a single, more easily searchable, page.
That admin guide is *still* too long, but this should help a bit.
Change-Id: I946c72f40dce2f33ef845a0ca816038727848b3a

Change-Id: Ibed9dc0afbdb922d06f7798bdac01db7c55b19f1

Change-Id: Idbff951506ee2f3b288eda00217c902314393877

Closes-Bug: #2016463
Change-Id: Id500a2429b7412823970a06e3e82b1d1646c70b8

Change-Id: I7ab605d48972e8dc06e630d160c745baeea91355

The OpenStack project is currently maintained on opendev.org, with github.com serving as a mirror repository.
Replace the source code repository address for the python-swiftclient project from github.com to opendev.org.
Change-Id: I650a80cb45febc457c42360061faf3a9799e6131

Reseller admins can set new headers on accounts like
 X-Account-Quota-Bytes-Policy-<policy-name>: <quota>
This may be done to limit consumption of a faster, all-flash policy, for
example.
This is independent of the existing X-Account-Meta-Quota-Bytes header, which
continues to limit the total storage for an account across all policies.
Change-Id: Ib25c2f667e5b81301f8c67375644981a13487cfe

Explain how to prepare venv with `tox devenv`
Closes-Bug: #2003984
Change-Id: Idc536034a36646de9c1880c8d0bc0a387b130ac2

Change-Id: I29d00e939a3842bd064382575955fa3e255242eb

nose has not seen active development for many years now. With py310, we
can no longer use it due to import errors.
Also update lower contraints
Closes-Bug: #1993531
Change-Id: I215ba0d4654c9c637c3b97953d8659ac80892db8

If you've got thousands of requests per second for objects in a single
container, you basically NEVER want that container's info to ever fall
out of memcache. If it *does*, all those clients are almost certainly
going to overload the container.
Avoid this by allowing some small fraction of requests to bypass and
refresh the cache, pushing out the TTL as long as there continue to be
requests to the container. The likelihood of skipping the cache is
configurable, similar to what we did for shard range sets.
Change-Id: If9249a42b30e2a2e7c4b0b91f947f24bf891b86f
Closes-Bug: #1883324 

Change-Id: I8ce6749c3d634c68e5d4a15d812a046514cc35f5

* Get rid of a bunch of accidental blockquote formatting
* Always declare a lexer to use for ``.. code::`` blocks
Change-Id: I8940e75b094843e542e815dde6b6be4740751813

* Call out liberasurecode as a requirement
* Include more py3 versions
* Use anonymous links
Change-Id: Ib1f8ef5e36825b9c241d2a4d838ea01b3df70da0

We've known this would eventually be necessary for a while [1], and
way back in 2017 we started seeing SHA-1 collisions [2].
This patch follows the approach of soft deprecation of SHA1 in tempurl.
It's still a default digest, but we'll start with warning as the
middleware is loaded and exposing any deprecated digests
(if they're still allowed) in /info.
Further, because there is much shared code between formpost and tempurl, this
patch also goes and refactors shared code out into swift.common.digest.
Now that we have a digest, we also move digest related code:
 - get_hmac
 - extract_digest_and_algorithm
[1] https://www.schneier.com/blog/archives/2012/10/when_will_we_se.html
[2] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
Change-Id: I581cadd6bc79e623f1dae071025e4d375254c1d9

Currently the object-replicator has an option called `handoff_delete`
which allows us to define the the number of replicas which are ensured
in swift. Once a handoff node ensures that many successful responses it
can go ahead and delete the handoff partition.
By default it's 'auto' or rather the number of primary nodes. But this
can be reduced. It's useful in draining full disks, but has to be used
carefully.
This patch adds the same option to the DB replicator and works the same
way. But instead of deleting a partition it's done at the per DB level.
Because it's done in the DB Replicator level it means the option is now
available to both the Account and Container replicators.
Change-Id: Ide739a6d805bda20071c7977f5083574a5345a33

Change-Id: Id441688aac1088041e243b8ee70710d9c5d7911b

When switching the s3api cross-compatibility tests' target between a
Swift endpoint and an S3 endpoint, allow specifying an AWS CLI style
credentials file as an alternative to editing the swift 'test.conf'
file.
Change-Id: I5bebca91821552d7df1bc7fa479b6593ff433925

This is a fairly blunt tool: ratelimiting is per device and
applied independently in each worker, but this at least provides
some limit to disk IO on backend servers.
GET, HEAD, PUT, POST, DELETE, UPDATE and REPLICATE methods may be
rate-limited.
Only requests with a path starting '<device>/<partition>', where
<partition> can be cast to an integer, will be rate-limited. Other
requests, including, for example, recon requests with paths such as
'recon/version', are unconditionally forwarded to the next app in the
pipeline.
OPTIONS and SSYNC methods are not rate-limited. Note that
SSYNC sub-requests are passed directly to the object server app
and will not pass though this middleware.
Change-Id: I78b59a081698a6bff0d74cbac7525e28f7b5d7c1

Apparently we fixed that recently without realizing it.
Change-Id: I2f623ffc1400f018c203e930a7b78dfdb9d6e61c
Related-Change: I8c98791a920eeedfc79e8a9d83e5032c07ae86d3

We said this would be going away back in 1.7.0 -- lets actually remove it.
Change-Id: I9742dd907abea86da9259740d913924bb1ce73e7
Related-Change: Id7d6d547b103b4f23ebf5be98b88f09ec6027ce4

We've known this would eventually be necessary for a while [1], and
way back in 2017 we started seeing SHA-1 collisions [2].
[1] https://www.schneier.com/blog/archives/2012/10/when_will_we_se.html
[2] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
UpgradeImpact:
==============
"sha1" has been removed from the default set of `allowed_digests` in the
tempurl middleware config. If your cluster still has clients requiring
the use of SHA-1,
- explicitly configure `allowed_digests` to include "sha1" and
- encourage your clients to move to more-secure algorithms.
Depends-On: https://review.opendev.org/c/openstack/tempest/+/832771
Change-Id: I6e6fa76671c860191a2ce921cb6caddc859b1066
Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Closes-Bug: #1733634 

Replace github by opendev because currently opendev is the source and
github is its mirror.
Also, update links for repositories managed by SwiftStack organization.
Unfortunately some repositories are no longer available so are removed
from the list.
Change-Id: Ic223650eaf7a1934f489c8b713c6d8da1239f3c5

The swauth project is already retired[1]. The documentation is updated
to reflect status of the project.
Also, this change removes reference to this middleware in unit tests.
[1] https://opendev.org/x/swauth/
Change-Id: I3d8e46d85ccd965f9b51006c330e391dcdc24a34

Change-Id: I4ee5a52ec9fb5f1920cd6869f6b1245c3787391c

As part of that, the ceph test runner needed up-rev'ing to run under
py3. As a result, the known-failures shifted.
Trim the on-demand rolling upgrade jobs list -- now that it's running
py3, we only expect it to pass for train and beyond.
Also, pin smmap version on py2 -- otherwise, the remaining experimental
jobs running on centos-7 fail.
Change-Id: Ibe46aecf0f4461be59eb206bfe9063cc1bfff706

When deleting multiple objects, S3 allows to enable
a quiet mode with the 'Quiet' key.
At AWS S3, the value of this key is case-insensitive.
- Quiet mode is enabled if the value is 'true'
 (regardless of case).
- Otherwise, in all other cases (even a non-boolean value),
 this mode will be disabled.
Also, some tools (like Minio's python API) send the value 'True'
(and not 'true').
Change-Id: Id9d1da2017b8d13242ae1f410347febb013e9ce1

Drive-By: make the register_sensitive_header() implementation more
obviously case-insensitive.
Change-Id: I5b299bc0adb526c468c6364a5706eb86809533e5

Several headers and query params were previously revealed in logs but
are now redacted:
 * X-Auth-Token header (previously redacted in the {auth_token} field,
 but not the {headers} field)
 * temp_url_sig query param (used by tempurl middleware)
 * Authorization header and X-Amz-Signature and Signature query
 parameters (used by s3api middleware)
This patch adds some new middleware helper methods to track headers and
query parameters that should be redacted by proxy-logging. While
instantiating the middleware, authors can call either:
 register_sensitive_header('case-insensitive-header-name')
 register_sensitive_param('case-sensitive-query-param-name')
to add items that should be redacted. The redaction uses proxy-logging's
existing reveal_sensitive_prefix config option to determine how much to
reveal.
Note that query params will still be logged in their entirety if
eventlet_debug is enabled.
UpgradeImpact
=============
The reveal_sensitive_prefix config option now applies to more items;
operators should review their currently-configured value to ensure it
is appropriate for these new contexts. In particular, operators should
consider reducing the value if it is more than 20 or so, even if that
previously offered sufficient protection for auth tokens.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Closes-Bug: #1685798
Change-Id: I88b8cfd30292325e0870029058da6fb38026ae1a

Previously, the set_statsd_prefix method was used to mutate a logger's
StatsdClient tail prefix after a logger was instantiated. This pattern
had led to unexpected mutations (see Related-Change). The tail_prefix
can now be passed as an argument to get_logger(), and is then
forwarded to the StatsdClient constructor, for a more explicit
assignment pattern.
The set_statsd_prefix method is left in place for backwards
compatibility. A DeprecationWarning will be raised if it is used
to mutate the StatsdClient tail prefix.
Change-Id: I7692860e3b741e1bc10626e26bb7b27399c325ab
Related-Change: I0522b1953722ca96021a0002cf93432b973ce626