openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Hopefully if we never do a release that supports signalfx no one will
ever use it and we won't have to maintain it.
Drive-by: refactor label model dispatch to fix a weird bug where a
config name could be a class attribute and blow up weird.
Change-Id: I2c67b59820c5ca094077bf47628426f4b0445ba0

Introduce a LabeledStatsdClient API; no callers yet.
Include three config options:
 - statsd_label_mode, which specifies which label format to use
 - statsd_emit_legacy, which dictates whether to emit old-style
 metrics dotted metrics
 - statsd_user_label_<name> = <value>, which supports user defined
 labels in restricted ASCII characters
Co-Authored-By: yanxiao@nvidia.com
Co-Authored-By: alistairncoles@gmail.com
Change-Id: I115ffb1dc601652a979895d7944e011b951a91c1

The following configuration options are deprecated:
 * expiring_objects_container_divisor
 * expiring_objects_account_name
The upstream maintainers are not aware of any clusters where these have
been configured to non-default values.
UpgradeImpact:
Operators are encouraged to remove their "container_divisor" setting and
use the default value of 86400.
If a cluster was deployed with a non-standard "account_name", operators
should remove the option from all configs so they are using a supported
configuration going forward, but will need to deploy stand-alone expirer
processes with legacy expirer config to clean-up old expiration tasks
from the previously configured account name.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Jianjian Huo <jhuo@nvidia.com>
Change-Id: I5ea9e6dc8b44c8c5f55837debe24dd76be7d6248

Related-Change: I8227939d04fda8db66fb2f131f2c71ce8741c7d9
Change-Id: I149a2df2d942bba02049947865b000c9cf1a89bc

Add a new tunable, `stale_worker_timeout`, defaulting to 86400 (i.e. 24
hours). Once this time elapses following a reload, the manager process
will issue SIGKILLs to any remaining stale workers.
This gives operators a way to configure a limit for how long old code
and configs may still be running in their cluster.
To enable this, the temporary reload child (which waits for the reload
to complete then closes the accept socket on all the old workers) has
grown the ability to send state to the re-exec'ed manager. Currently,
this is limited to just the set of pre-re-exec child PIDs and their
reload times, though it was designed to be reasonably extensible.
This allows the new manager to recognize stale workers as they exit
instead of logging
 Ignoring wait() result from unknown PID ...
With the improved knowledge of subprocesses, we can kick the log level
for the above message up from info to warning; we no longer expect it
to trigger in practice.
Drive-by: Add logging to ServersPerPortStrategy.register_worker_exit
that's comparable to what WorkersStrategy does.
Change-Id: I8227939d04fda8db66fb2f131f2c71ce8741c7d9

Change-Id: Ic66b9ae89837afe31929ce07cc625dfc28314ea3

* Remove py2 gate jobs.
* Build non-universal, py3-only wheels.
* Specify minimum python version in package metadata.
* Clean up requirements/constraints/bindep (a little, anyway).
Change-Id: I53153c4fde043e964e1daa7bbf2089e0471dede2

Changed OS version from RHEL 7 and CentOS 7 to RHEL 9 and
CentOS Stream 9.
Changed python to python3.
Changed yum command to dnf command.
Change-Id: Ie1e815c0434255e77ef5e9103576f85d9d6490ae

In order to modernize swift's statsd configuration we're working to
separate it from logging. This change is a pre-requisite for the
Related-Change in order to simplfy the stdlib base logger instance
wrapping in a single extended SwiftLogAdapter (previously LogAdapter)
which supports all the features swift's servers/daemons need
from our logger instance interface.
Related-Change-Id: I44694b92264066ca427bb96456d6f944e09b31c0
Change-Id: I8988c0add6bb4a65cc8be38f0bf527f141aac48a

Drive-Bys:
 * DRY out redundent configuration examples in expiring objects overview
 documentation.
 * Add missing delay_reaping man page docs.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I8879dbd13527233c878dff764ec411ce9619ee39

Some submodules have previously been broken out of the utils
module. This patch adds automodule directives for the new modules to
the source documentation.
Change-Id: I985205fda95f01d226e81dcbfe0d6dbbb5b69c96
Related-Change: Ic4b5005e3efffa8dba17d91a41e46d5c68533f9a

The datetime.utcfromtimestamp() is deprecated in Python 3.12.
Replace datetime.utcfromtimestamp() with datetime.fromtimestamp().
Change-Id: I01d6b94de394413aa13a045ab2c36504e65a6f5a
Signed-off-by: Takashi Natsume <takanattie@gmail.com>

Those were so unwieldy as to be uneditable before.
Change-Id: Ic9f4a0ea6b8e18e1624c516890ab69884a299773

For some reason, when we switched from py36 on centos8 to py39 on
centos9, these two tests started failing. Looks like a disagreement
about whether the canonical path for a bucket request should have
a trailing slash or not.
Mark them as known-failures for now so we can stay aware of any
other new breakage brought on by swift code changes.
Related-Change: I4f6b9c07af7bc768654f1a5d0c66b048e0f2c9c1
Change-Id: If990752c7ef7667182dbe18e49679e48c0e3d42d

The old one now just redirects to https://developer.adobe.com/
Change-Id: Iba281bb8dceedef9bbb702abf0b93473bf91d9f7

If the global configuration option 'enable_open_expired' is set
to true in the config, then the client will be able to make a
request with the header 'x-open-expired' set to true in order
to access an object that has expired, provided it is in its
grace period. If this config flag is set to false, the client
will not be able to access any expired objects, even with the
header, which is the default behavior unless the flag is set.
When a client sets a 'x-open-expired' header to a true value for a
GET/HEAD/POST request the proxy will forward x-backend-open-expired to
storage server. The storage server will allow clients that set
x-backend-open-expired to open and read an object that has not yet
been reaped by the object-expirer, even after the x-delete-at time
has passed.
The header is always ignored when used with temporary URLs.
Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Related-Change: I106103438c4162a561486ac73a09436e998ae1f0
Change-Id: Ibe7dde0e3bf587d77e14808b169c02f8fb3dddb3

The object expirer can be configured to delay the reaping of
objects from disk after their expiration time using account
and container level delay_reaping values. The delay_reaping
value of accounts and containers in seconds is configured in
the object server config. The object expirer references these
configured values to only reap objects from specified accounts
and containers after their corresponding delays.
The goal of the delay_reaping feature is to prevent accidental or
premature data loss if an object marked for deletion with the
'x-delete-at' feature should not be reaped immediately, for
whatever reason.
Configuring the delay_reaping value at a granular account and
container level is beneficial for being able to keep storage
capacity consumption in control while maintaining a desired
data recovery window.
This patch also adds a sample configuration, documentation, and
tests for bad configurations and grace period functionality.
Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Change-Id: I106103438c4162a561486ac73a09436e998ae1f0

Change-Id: I2aa13e2b23bda86c51ef6aaa69ea3fd0075bb9ad

When logging a request, if the request environ has a
swift.proxy_logging_status item then use its value for the log
message status int.
The swift.proxy_logging_status hint may be used by other middlewares
when the desired logged status is different from the wire_status_int.
If the proxy_logging middleware detects a client disconnect then any
swift.proxy_logging_status item is ignored and a 499 status int is
logged, as per current behaviour. i.e.:
 * client disconnect overrides swift.proxy_logging_status and the
 response status
 * swift.proxy_logging_status overrides the response status
If the proxy_logging middleware catches an exception then the logged
status int will be 500 regardless of any swift.proxy_logging_status
item.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I9b5cc6d5fb69a2957b8c4846ce1feed8c115e6b6

The per-service option was deprecated almost 4 years ago[1].
[1] 4601548dab
Change-Id: I45f7678c9932afa038438ee841d1b262d53c9da8

Talk about graceful shutdowns, seamless reloads, and the swift-reload helper.
Change-Id: I9c86d1ce9853840d73e2107a6d79b2454c21438a

Related-Change: Icff8cf57474dfad975a4f45bf2d500c2682c1129
Change-Id: Ic2283fab0d18ea03c6beb353c6b934344606c15e

Some tooling out there, like Ansible, will always call to see if
object-lock is enabled on a bucket/container. This fails as Swift doesn't
understand the object-lock or the get object lock api[0].
When you use the get-object-lock-configuration to a bucket in s3 that
doesn't have it applied it returns a specific 404:
 GET /?object-lock HTTP/1.1" 404 None
 ...
 <?xml version="1.0" encoding="UTF-8"?>
 <Error>
 <Code>ObjectLockConfigurationNotFoundError</Code>
 <Message>Object Lock configuration does not exist for this bucket</Message>
 <BucketName>bucket_name</BucketName>
 <RequestId>83VQBYP0SENV3VP4</RequestId>
 </Error>'
This patch doesn't add support for get_object lock, instead it always
returns a similar 404 as supplied by s3, so clients know it's not
enabled.
Also add a object-lock PUT 501 response.
[0] https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html
Change-Id: Icff8cf57474dfad975a4f45bf2d500c2682c1129

If swift is used with an Application Credential with access_rules [1],
the catalog needs to be sent with the token for it to be evaluated by
keystonemiddleware.
[1]: https://docs.openstack.org/keystone/2023.1/user/application_credentials.html#access-rules
Change-Id: I0b02821722ffc819bc8a97281a33391072242ffc

Change-Id: I6ef909e826d3901f24d3c42a78d2ab1e4e47bb64

Personally I'm not a big fan of how we arrange logs for SAIO,
but it is a historic standard. The reconciler has to conform.
Change-Id: I45a25ff406b31b6b1b403e213554aaabfebc6eb5

The never ending changes in package names march on.
This is for Fedora 38.
Change-Id: Idffd1df7e2f23681eb7d7bf1717471b594612dbc

Change-Id: I5e75a359660315a34ec36f5cb748808257f4185a

Currently, SLO manifest files will be evicted from page cache
after reading it, which cause hard drives very busy when user
requests a lot of parallel byte range GETs for a particular
SLO object.
This patch will add a new config 'keep_cache_slo_manifest', and
try keeping the manifest files in page cache by not evicting them
after reading if config settings allow so.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I557bd01643375d7ad68c3031430899b85908a54f

Change-Id: I04478c44957032e911069bcd4859caf3baaddb63

Change-Id: I6fc9eaf0903bdbe4ccb82c4e7647478a1eef9785

Offer it both by service and as a single, more easily searchable, page.
That admin guide is *still* too long, but this should help a bit.
Change-Id: I946c72f40dce2f33ef845a0ca816038727848b3a

Change-Id: Ibed9dc0afbdb922d06f7798bdac01db7c55b19f1

Change-Id: Idbff951506ee2f3b288eda00217c902314393877

Closes-Bug: #2016463
Change-Id: Id500a2429b7412823970a06e3e82b1d1646c70b8

Change-Id: I7ab605d48972e8dc06e630d160c745baeea91355

The OpenStack project is currently maintained on opendev.org, with github.com serving as a mirror repository.
Replace the source code repository address for the python-swiftclient project from github.com to opendev.org.
Change-Id: I650a80cb45febc457c42360061faf3a9799e6131

Reseller admins can set new headers on accounts like
 X-Account-Quota-Bytes-Policy-<policy-name>: <quota>
This may be done to limit consumption of a faster, all-flash policy, for
example.
This is independent of the existing X-Account-Meta-Quota-Bytes header, which
continues to limit the total storage for an account across all policies.
Change-Id: Ib25c2f667e5b81301f8c67375644981a13487cfe

Explain how to prepare venv with `tox devenv`
Closes-Bug: #2003984
Change-Id: Idc536034a36646de9c1880c8d0bc0a387b130ac2

Change-Id: I29d00e939a3842bd064382575955fa3e255242eb