openstack/swift - swift - OpenDev: Free Software Needs Free Tools

- swift-account-audit
 - swift-config
 - swift-drive-audit
 - swift-get-nodes
 - swift-reconciler-enqueue
 - swift-oldies
 - swift-orphans
Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: Ie4a252ec51d23caa9e90e6c2f772847d0d71b1c8

Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: I2718d8b96193d13f913824fc1091d72737d755ea

Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: I7f479359f94f102414f30e8aac87f377d10d34ee

I *think* swift.common.manager is a reasonable place for it?
Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: I48a345eedbf2369d481d18c0e2db37845673b647

See https://review.opendev.org/c/openstack/swift/+/918365 for motivation.
A handful of [files]scripts entries still remain, but they're more
involved to change over. These ones were all fairly mechanical to fix.
Related-Change: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83
Change-Id: Ia43d7cd3921bc6c9ff0cee3100ef5f486fd3edcb

Previously, `twine check <swift wheel>` would report
 WARNING `long_description_content_type` missing.
 defaulting to `text/x-rst`.
Change-Id: I5d004218515d17b8cbace46b2c1e646f734779f3

Change-Id: If15c3fee706d295de678f812839a5d0759089ccf

The old [files]scripts method of specifying executable Python scripts
triggers some legacy easy-install-like mode for editable installs,
which relies on pkg_resources. Recent versions of setuptools (67.5.0+)
have started emitting warnings when importing pkg_resources, which in
turn cause quite noticeable slowdowns in process startup. This is
particularly prominant on py312, which stopped pre-installing (an often
older version of) setuptools in new venvs.
See also:
 - https://github.com/python/cpython/issues/95299
 - https://github.com/pypa/setuptools/pull/3843
 - https://github.com/pypa/setuptools/issues/3966
Now, use [entry_points]console_scripts to specify these executables,
which does not use pkg_resources in the generated script files.
Change-Id: Ifcc8138e7b55d5b82bea0d411ec6bfcca2c77c83

Change-Id: I52153d27f53b81de63b6b3590e1ab22f563f178f

As per the current release tested runtime, we test
python version from 3.8 to 3.11 so updating the
same in python classifier in setup.cfg
Change-Id: If270236c2b23f432e1a3e1508101a7cc86bbd73b

Previously, WSGI server systemd unit files might have used something like
 ExecReload=kill -USR1 $MAINPID
This was risky; in the related change, reloads were made safer, but
required more than one ExecReload line. Meanwhile, systemd docs
(https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecReload=)
say
> It is strongly recommended to set ExecReload= to a command that
> not only triggers a configuration reload of the daemon, but also
> synchronously waits for it to complete.
which *neither* set of ExecReloads would do.
Now, add a new swift-reload command which, given a pid,
 * validates that the PID seems to belong to a Swift WSGI server
 manager process,
 * checks that the config used by that PID is still valid,
 * signals the PID to perform a seamless reload, and
 * waits for the reload to complete by monitoring the PID's children.
As a result, WSGI server systemd unit files can now use something like
 ExecReload=swift-reload $MAINPID
to follow systemd recommendations.
Change-Id: Ifcadd2f8427f107aae1921cdd311f7973b0312e1
Related-Change: I9e5e158ce8be92535430b9cabf040063f5188bf4

Change-Id: I262b6c8fbe01bd4e08e9c685fc61b429ccc2d7e3

Change-Id: I570e1b00ba0f68af5e6c66933412d61f57f663e6

- Switch to using `url` instead of the `home_page` alias
- Add links to docs, LP, source, and changelog
Change-Id: I5590b9d7402c9d128278f92fd522bf7364d781a4

We've been running py310 unit tests for a while, and with DSVM on jammy,
we've now got py310 func tests.
Change-Id: Iec8c8ee6d5dc4fec169c6d2367053fb1469f45f0

This is a fairly blunt tool: ratelimiting is per device and
applied independently in each worker, but this at least provides
some limit to disk IO on backend servers.
GET, HEAD, PUT, POST, DELETE, UPDATE and REPLICATE methods may be
rate-limited.
Only requests with a path starting '<device>/<partition>', where
<partition> can be cast to an integer, will be rate-limited. Other
requests, including, for example, recon requests with paths such as
'recon/version', are unconditionally forwarded to the next app in the
pipeline.
OPTIONS and SSYNC methods are not rate-limited. Note that
SSYNC sub-requests are passed directly to the object server app
and will not pass though this middleware.
Change-Id: I78b59a081698a6bff0d74cbac7525e28f7b5d7c1

We've been seeing warnings like
 UserWarning: Usage of dash-separated 'home-page' will not be supported
 in future versions. Please use the underscore name 'home_page' instead
for a while in our pep8 jobs; this ought to clean them up. As best I can
tell, setuptools has supported either format for several years.
Change-Id: I4d538859f278ac26a6b6d6845df103ebd4c49da3

Depends-On: https://review.opendev.org/c/openstack/project-config/+/774906
Change-Id: I4a16370646a1a684240d29950ba197baccae91bb

Swift operators may find it useful to operate on each object in their
cluster in some way. This commit provides them a way to hook into the
object auditor with a simple, clearly-defined boundary so that they
can iterate over their objects without additional disk IO.
For example, a cluster operator may want to ensure a semantic
consistency with all SLO segments accounted in their manifests,
or locate objects that aren't in container listings. Now that Swift
has encryption support, this could be used to locate unencrypted
objects. The list goes on.
This commit makes the auditor locate, via entry points, the watchers
named in its config file.
A watcher is a class with at least these four methods:
 __init__(self, conf, logger, **kwargs)
 start(self, audit_type, **kwargs)
 see_object(self, object_metadata, data_file_path, **kwargs)
 end(self, **kwargs)
The auditor will call watcher.start(audit_type) at the start of an
audit pass, watcher.see_object(...) for each object audited, and
watcher.end() at the end of an audit pass. All method arguments are
passed as keyword args.
This version of the API is implemented on the context of the
auditor itself, without spawning any additional processes.
If the plugins are not working well -- hang, crash, or leak --
it's easier to debug them when there's no additional complication
of processes that run by themselves.
In addition, we include a reference implementation of plugin for
the watcher API, as a help to plugin writers.
Change-Id: I1be1faec53b2cdfaabf927598f1460e23c206b0a

Remove babel.cfg and the translation bits from setup.cfg, those are not
needed anymore.
Change-Id: I444e018b52279ffed96158a96cf66123a0e6267b

Change-Id: Icd5312dec647f69a5df111c3a6766093ecfcb6ef

Users have complained for a while that Swift's ETags don't match the
expected RFC formats. We've resisted fixing this for just as long,
worrying that the fix would break innumerable clients that expect the
value to be a hex-encoded MD5 digest and *nothing else*.
But, users keep asking for it, and some consumers (including some CDNs)
break if we *don't* have quoted etags -- so, let's make it an option.
With this middleware, Swift users can set metadata per-account or even
per-container to explicitly request RFC compliant etags or not. Swift
operators also get an option to change the default behavior
cluster-wide; it defaults to the old, non-compliant format.
See also:
 - https://tools.ietf.org/html/rfc2616#section-3.11
 - https://tools.ietf.org/html/rfc7232#section-2.3
Closes-Bug: 1099087
Closes-Bug: 1424614
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I380c6e34949d857158e11eb428b3eda9975d855d

Apparently this had been done for us previously? See also:
http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012097.html
Change-Id: I27288f9e503f26827a8274d46b07c1330a21de10

This is a mechanically generated patch to ensure unit testing is in place
for all of the Tested Runtimes for Train.
See the Train python3-updates goal document for details:
https://governance.openstack.org/tc/goals/train/python3-updates.html
Change-Id: I29f2951053fa6714de46ed757116d99dd5c896cf
Story: #2005924
Task: #34249 

Adds a tool, swift-container-deleter, that takes an account/container
and optional prefix, marker, and/or end-marker; spins up an internal
client; makes listing requests against the container; and pushes the
found objects into the object-expirer queue with a special
application/async-deleted content-type.
In order to do this enqueuing efficiently, a new internal-to-the-cluster
container method is introduced: UPDATE. It takes a JSON list of object
entries and runs them through merge_items.
The object-expirer is updated to look for work items with this
content-type and skip the X-If-Deleted-At check that it would normally
do.
Note that the target-container's listing will continue to show the
objects until data is actually deleted, bypassing some of the concerns
raised in the related change about clearing out a container entirely and
then deleting it.
Change-Id: Ia13ee5da3d1b5c536eccaadc7a6fdcd997374443
Related-Change: I50e403dee75585fc1ff2bb385d6b2d2f13653cf8

Change-Id: Ib5f0b7431819892510d1913465ce4a81c92bdf25

With this commit, each storage policy can define the diskfile to use to
access objects. Selection of the diskfile is done in swift.conf.
Example:
 [storage-policy:0]
 name = gold
 policy_type = replication
 default = yes
 diskfile = egg:swift#replication.fs
The diskfile configuration item accepts the same format than middlewares
declaration: [[scheme:]egg_name#]entry_point
The egg_name is optional and default to "swift". The scheme is optional
and default to the only valid value "egg". The upstream entry points are
"replication.fs" and "erasure_coding.fs".
Co-Authored-By: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I070c21bc1eaf1c71ac0652cec9e813cadcc14851

Add a new middleware that can be used to fetch an encryption root
secret from a KMIP service. The middleware uses a PyKMIP client
to interact with a KMIP endpoint. The middleware is configured with
a unique identifier for the key to be fetched and options required
for the PyKMIP client.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ib0943fb934b347060fc66c091673a33bcfac0a6d

Provides a simple, experimental, CLI tool to generate a
composite ring from a list of component builder files.
For example:
 swift-ring-composer <composite-file> compose \
 <builder-file> <builder-file> --output <ring-file>
Commands available:
- compose: compose a list of builder file to a composite ring
- show: show the metadata for a composite ring
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: I25a79e71c13af352e19e4358f60545265b51584f

This patch adds a read_only middleware to swift. It gives the ability
to make an entire cluster or individual accounts read only.
When a cluster or an account is in read only mode, requests that would
result in writes to the cluser are not allowed.
DocImpact
Change-Id: I7e0743aecd60b171bbcefcc8b6e1f3fd4cef2478

The sharder daemon visits container dbs and when necessary executes
the sharding workflow on the db.
The workflow is, in overview:
- perform an audit of the container for sharding purposes.
- move any misplaced objects that do not belong in the container
 to their correct shard.
- move shard ranges from FOUND state to CREATED state by creating
 shard containers.
- move shard ranges from CREATED to CLEAVED state by cleaving objects
 to shard dbs and replicating those dbs. By default this is done in
 batches of 2 shard ranges per visit.
Additionally, when the auto_shard option is True (NOT yet recommeneded
in production), the sharder will identify shard ranges for containers
that have exceeded the threshold for sharding, and will also manage
the sharding and shrinking of shard containers.
The manage_shard_ranges tool provides a means to manually identify
shard ranges and merge them to a container in order to trigger
sharding. This is currently the recommended way to shard a container.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7f192209d4d5580f5a0aa6838f9f04e436cf6b1f

This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.
1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)
2. Port unittests to test/unit/s3api dir to be able to run on the gate.
3. Port functests to test/functional/s3api and setup in-process testing
4. Port docs to doc dir, then address the namespace change.
5. Use get_logger() instead of global logger instance
6. Avoid global conf instance
Ex. fix various minor issue on those steps (e.g. packages, dependencies,
 deprecated things)
The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)
Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.
Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>

For compliance with the Project Testing Interface as described in:
https://governance.openstack.org/tc/reference/project-testing-interface.html
For more details information, please refer to:
http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html
Co-Authored-By: Nguyen Hai <nguyentrihai93@gmail.com>
Change-Id: I26dc41c7df57bf79db531c6e67e148e01c17e992

Change-Id: I805cf220aee4af69bf4984cc148d69520e958073

Added auth_url to the context we pass to castellan
library. In a change [1] intended to deprecate the use
of auth_endpoint passed as the oslo config, it actually
completely removed the use of it[2], so this change became
necessary or the integration is broken.
[1] - https://review.openstack.org/#/c/483457
[2] - https://review.openstack.org/#/c/483457/6/castellan/key_manager/barbican_key_manager.py@143
Change-Id: I933367fa46aa0a3dc9aedf078b1be715bfa8c054

Add a symbolic link ("symlink") object support to Swift. This
object will reference another object. GET and HEAD
requests for a symlink object will operate on the referenced object.
DELETE and PUT requests for a symlink object will operate on the
symlink object, not the referenced object, and will delete or
overwrite it, respectively.
POST requests are *not* forwarded to the referenced object and should
be sent directly. POST requests sent to a symlink object will
result in a 307 Error.
Historical information on symlink design can be found here:
https://github.com/openstack/swift-specs/blob/master/specs/in_progress/symlinks.rst.
https://etherpad.openstack.org/p/swift_symlinks
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Signed-off-by: Thiago da Silva <thiago@redhat.com>

This has been deprecated since Swift 2.10.0 (Newton) including a
message that it would go away. Let's actually remove it.
Change-Id: I7d3659761c71119363ff2c0c750e37b4c6374a39
Related-Change: Ifa8bf636f20f82db4845b02d1b58699edaa39356

Make some json -> (text, xml) stuff in a common module, reference that in
account/container servers so we don't break existing clients (including
out-of-date proxies), but have the proxy controllers always force a json
listing.
This simplifies operations on listings (such as the ones already happening in
decrypter, or the ones planned for symlink and sharding) by only needing to
consider a single response type.
There is a downside of larger backend requests for text/plain listings, but
it seems like a net win?
Change-Id: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d

Change-Id: I81736080fc478c2b69d5b71edd0cada39aad9400

This patch allows to import the dispersion report tool, and thus making
it more easily usable within other Python tools. This can be also used
in a follow up patch to add some tests for the report tool.
It also fixes a bug when using the "--dump-json" option - until now it
returned the policy name and made the JSON invalid.
Change-Id: Ie0d52a1a54fc152bb72cbb3f84dcc36a8dad972a

This patch adds support for retrieving the encryption root secret from
an external key management system. In practice, this is currently
limited to Barbican.
Change-Id: I1700e997f4ae6fa1a7e68be6b97539a24046e80b

* Fixes warnings in RST file
* Suppress warning log from pyeclib during the doc build.
 pyeclib emits a warning message on an older liberasurecode [1]
 and sphinx treats this as error (when warning-is-error is set).
 There is no need to check warnings during the doc build,
 so we can safely suppress the warning.
This is a part of the doc migration community-wide effort.
http://specs.openstack.org/openstack/docs-specs/specs/pike/os-manuals-migration.html
[1] https://github.com/openstack/pyeclib/commit/d163972b
Change-Id: I9adaee29185a2990cc3985bbe0dd366e22f4f1a2

Change-Id: I9590d129e7d5e72d955983f3ce38b4e2d44c95b1

This patch adds methods to increase the partition power of an existing
object ring without downtime for the users using a 3-step process. Data
won't be moved to other nodes; objects using the new increased partition
power will be located on the same device and are hardlinked to avoid
data movement.
1. A new setting "next_part_power" will be added to the rings, and once
the proxy server reloaded the rings it will send this value to the
object servers on any write operation. Object servers will now create a
hard-link in the new location to the original DiskFile object. Already
existing data will be relinked using a new tool in the new locations
using hardlinks.
2. The actual partition power itself will be increased. Servers will now
use the new partition power to read from and write to. No longer
required hard links in the old object location have to be removed now by
the relinker tool; the relinker tool reads the next_part_power setting
to find object locations that need to be cleaned up.
3. The "next_part_power" flag will be removed.
This mostly implements the spec in [1]; however it's not using an
"epoch" as described there. The idea of the epoch was to store data
using different partition powers in their own namespace to avoid
conflicts with auditors and replicators as well as being able to abort
such an operation and just remove the new tree. This would require some
heavy change of the on-disk data layout, and other object-server
implementations would be required to adopt this scheme too.
Instead the object-replicator is now aware that there is a partition
power increase in progress and will skip replication of data in that
storage policy; the relinker tool should be simply run and afterwards
the partition power will be increased. This shouldn't take that much
time (it's only walking the filesystem and hardlinking); impact should
be low therefore. The relinker should be run on all storage nodes at the
same time in parallel to decrease the required time (though this is not
mandatory). Failures during relinking should not affect cluster
operations - relinking can be even aborted manually and restarted later.
Auditors are not quarantining objects written to a path with a different
partition power and therefore working as before (though they are reading
each object twice in the worst case before the no longer needed hard
links are removed).
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
[1] https://specs.openstack.org/openstack/swift-specs/specs/in_progress/
increasing_partition_power.html
Change-Id: I7d6371a04f5c1c4adbb8733a71f3c177ee5448bb