openstack/swift - swift - OpenDev: Free Software Needs Free Tools

There are times when it is convenient to specify a policy by name or
by index (see Related-Change), but policy names can unfortunately
collide with indexes. Using a number as a policy name should at least
be discouraged.
Change-Id: I0cdd3b86b527d6656b7fb50c699e3c0cc566e732
Related-Change: Icf1517bd930c74e9552b88250a7b4019e0ab413e

Change-Id: I799bdacb7ef5850096c2178bfb12959f4205920d

Unfortunately, we can't identify the user, so we can't map to an
account, so we can't respect whatever CORS metadata might be set on the
container.
As a result, the allowed origins must be configured cluster-wide. Add a
new config option, cors_preflight_allow_origin, for that; default it
to blank (ie, deny preflights from all origins, preserving existing
behavior), but allow either a comma-separated list of origins or
* (to allow all origins).
Change-Id: I985143bf03125a05792e79bc5e5f83722d6431b3
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>

When I tried to follow documentation, the expirer said:
 object-expirer[391675]: This node is not configured to dequeue
 tasks from the legacy queue.
Our documentation was incorrect, the actual name of the option is
"dequeue_from_legacy".
Change-Id: I5ca7ac589a405d0b6250922aa9bcaabecb3c4fb0

While working on the shrinking recon drops, we want to display numbers
that directly relate to how tool should behave. But currently all
options of the s-m-s-r tool is driven by cli options.
This creates a disconnect, defining what should be used in the sharder
and in the tool via options are bound for failure. It would be much
better to be able to define the required default options for your
environment in one place that both the sharder and tool could use.
This patch does some refactoring and adding max_shrinking and
max_expanding options to the sharding config. As well as adds a
--config option to the tool.
The --config option expects a config with at '[container-sharder]'
section. It only supports the shard options:
 - max_shrinking
 - max_expanding
 - shard_container_threshold
 - shard_shrink_point
 - shard_merge_point
The latter 2 are used to generate the s-m-s-r's:
 - shrink_threshold
 - expansion_limit
 - rows_per_shard
Use of cli arguments take precedence over that of the config.
Change-Id: I4d0147ce284a1a318b3cd88975e060956d186aec

Change-Id: I33c0168c1bb89f780be6fc317a4df89322cbc28d

Previously, we would rely on replication/reconstruction to rehash
once the part power increase was complete. This would lead to large
I/O spikes if operators didn't proactively tune down replication.
Now, do the rehashing in the relinker as it completes work. Operators
should already be mindful of the relinker's I/O usage and are likely
limiting it through some combination of cgroups, ionice, and/or
--files-per-second.
Also clean up empty partitions as we clean up. Operators with metrics on
cluster-wide primary/handoff partition counts can use them to monitor
relinking/cleanup progress:
 P 3N .----------.
 a / '.
 r / '.
 t 2N / .-----------------
 i / |
 t / |
 i N -----'---------'
 o
 n
 s 0 ----------------------------------
 t0 t1 t2 t3 t4 t5
At t0, prior to relinking, there are
 N := <replica count> * 2 ** <old part power>
primary partitions throughout the cluster and a negligible number of
handoffs.
At t1, relinking begins. In any non-trivial, low-replica-count cluster,
the probability of a new-part-power partition being assigned to the same
device as its old-part-power equivalent is low, so handoffs grow while
primaries remain constant.
At t2, relinking is complete. The total number of partitions is now 3N
(N primaries + 2N handoffs).
At t3, the ring with increased part power is distributed. The notion of
what's a handoff and what's a primary inverts.
At t4, cleanup begins. The "handoffs" are cleaned up as hard links and
now-empty partitions are removed.
At t5, cleanup is complete and there are now 2N total partitions.
Change-Id: Ib5bf426cf38559091917f2d25f4f60183cd16354

Swap out the standard logger stuff in place of --logfile. Keep --device
as a CLI-only option. Everything else is pretty standard stuff that
ought to be in [DEFAULT].
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I32f979f068592eaac39dcc6807b3114caeaaa814

Change-Id: I561ca419104bfb4b64affc477aac53a2fb98137b

UpgradeImpact
=============
Operators should verify that encryption is not enabled in their
reconciler pipelines; having it enabled there may harm data durability.
For more information, see https://launchpad.net/bugs/1910804
Change-Id: I1a1d78ed91d940ef0b4eba186dcafd714b4fb808
Closes-Bug: #1910804 

When you start getting more then 3k shards in a root container the
cached shard range listing can get bigger then the default max size for
memcache (1MB).
So add a mention about it in the configuration guide.
Fixes bug 1890643
Change-Id: If380410c17ed9ebc014b8198af0ea8d502deacc8

Also fix link to python paste.
Closes-Bug: #1734740
Closes-Bug: #1719887
Change-Id: I27ff441ed746f2919bcf9ca9a77c26371b18540b

We have a separate doc page for middlewares that pulls the docstring
from each middleware's docstring[0]. This makes it easy to look up the docs
in our documentation and easy to find the middleware doc by looking in
the code of the middleware itself.
This patch does the same with the audit watchers. There is now a page
that generates a list of audit watchers, even though currently it's only
one, and pulls the docs from their docstrings. Giving us an easy way to
maintain each audit watcher doc along with it's code.
[0] - https://docs.openstack.org/swift/latest/middleware.html
Change-Id: I1456aba0158d29fa0a879dcc2dfb13245c45ad16

Swift operators may find it useful to operate on each object in their
cluster in some way. This commit provides them a way to hook into the
object auditor with a simple, clearly-defined boundary so that they
can iterate over their objects without additional disk IO.
For example, a cluster operator may want to ensure a semantic
consistency with all SLO segments accounted in their manifests,
or locate objects that aren't in container listings. Now that Swift
has encryption support, this could be used to locate unencrypted
objects. The list goes on.
This commit makes the auditor locate, via entry points, the watchers
named in its config file.
A watcher is a class with at least these four methods:
 __init__(self, conf, logger, **kwargs)
 start(self, audit_type, **kwargs)
 see_object(self, object_metadata, data_file_path, **kwargs)
 end(self, **kwargs)
The auditor will call watcher.start(audit_type) at the start of an
audit pass, watcher.see_object(...) for each object audited, and
watcher.end() at the end of an audit pass. All method arguments are
passed as keyword args.
This version of the API is implemented on the context of the
auditor itself, without spawning any additional processes.
If the plugins are not working well -- hang, crash, or leak --
it's easier to debug them when there's no additional complication
of processes that run by themselves.
In addition, we include a reference implementation of plugin for
the watcher API, as a help to plugin writers.
Change-Id: I1be1faec53b2cdfaabf927598f1460e23c206b0a

Add a new config option to SLO, allow_async_delete, to allow operators
to opt-in to this new behavior. If their expirer queues get out of hand,
they can always turn it back off.
If the option is disabled, handle the delete inline; this matches the
behavior of old Swift.
Only allow an async delete if all segments are in the same container and
none are nested SLOs, that way we only have two auth checks to make.
Have s3api try to use this new mode if the data seems to have been
uploaded via S3 (since it should be safe to assume that the above
criteria are met).
Drive-by: Allow the expirer queue and swift-container-deleter to use
high-precision timestamps.
Change-Id: I0bbe1ccd06776ef3e23438b40d8fb9a7c2de8921

This patch moves the tables describing configuration options for each
server type from the deployment_guide.rst doc to separate per-server
documents. The new per-server documents are grouped under a config
directory with a config index doc. The config index doc is listed in
the top level index and provides a single starting point to navigate
to the individual server docs.
Change-Id: I6cedd98586febb5dc949c088ee44e160385ed324

We sometimes see probe test failures in the gate where `resetswift`
fails like
 umount: /mnt/sdb1: target is busy.
It seems to be because `swift-init all kill` is just a stop with a
status check after the fact. Add an orphan sweep afterwards to make
sure worker subprocesses got cleaned up, too.
Change-Id: I533b436a35e5c3deb7488e9961b55f2274c14751

Capture the on the wire status code for logging because we change the
logged status code sometimes.
Closes-Bug: #1896518
Change-Id: I27feabe923a6520e983637a9c68a19ec7174a0df

docs: Removing the use of NameVirtualHost from the apache examples
It's not used anymore. It's deprecated in fact: https://httpd.apache.org/docs/2.4/mod/core.html#namevirtualhost
Change-Id: I76999cfacc10a244024ee0cca66dda95a0169a67
docs: Added more spacing to the apache2 examples
They're easier to read and a bit less bloated.
Change-Id: I5e21a66018b7ef309918fbbde93f2494286d291e
docs: Switching to /srv/www to be more FHS 3.0 conformat
It's more modern and well supported to use /srv/www now in place of
/var/www.
Change-Id: Icd09ed4d5fb4e2b9b84ddead21313ea1c0a87c91
ref: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s17.html
docs: Added user, group and display name in WSGI examples
This properly sets the user and group for the wsgi processes in the
examples; as well as adding a display name for easier identification.
Change-Id: Ie5783081e4054e5b2fbf3a856716101a1aaf61b8
docs: Replace apachectl for systemctl commands
It's safe to asume that all modern distros; supported by OpenStack, will
have systemd implemented. It's better to favor systemctl in those cases.
Change-Id: Ic0d2e47c1ac53502ce638d6fc2424ab9df037262
docs: Emphasis to file paths and command options
I've enclosed configuration options or parameters in interpreted text
quotes.
Also, I've enclosed fiel paths with inline literal quotes.
Change-Id: Iec54b7758bce01fc8e8daff48498383cb70c62ce
docs: Fixed wording used to indicate the restart of apache
Just a little commit to make it clearer of what we're gonna do.
Change-Id: Id5ab3e94519bcfe1832b92e456a1d1fa81dd54e3

It's more modern and well supported to use /srv/www now in place of
/var/www.
Change-Id: Icd09ed4d5fb4e2b9b84ddead21313ea1c0a87c91
ref: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s17.html 

Change-Id: Ic07b0bdd5f1c90e7e25991def58c587680d7bfab

It does *not* include transfer time; it's just the time until response
is started.
Change-Id: I67c5c37c8c669e1e2de2cbfca8bcec02dc98659f

Note that existing SAIOs with 60xx ports should still work fine.
Change-Id: If5dd79f926fa51a58b3a732b212b484a7e9f00db
Related-Change: Ie1c778b159792c8e259e2a54cb86051686ac9d18

Lower the part-power -- 18 is way higher than is needed for a dev
environment.
Add commands for reduced-redundancy and erasure-coded storage policies.
Related-Change: Ibe46011d8e6a6482d39b3a20ac9c091d9fbc6ef7
Related-Change: I6f11f7a1bdaa6f3defb3baa56a820050e5f727f1
Related-Change: I0403016a4bb7dad9535891632753b0e5e9d402eb
Change-Id: I13de27674c81977c2470d43bbb2126ecc4bdd85a

My customers use the upstream documentation for manual
installation of Swift nodes and they come upon the lack
of any mention that host firewall ought to accomodate
services running on the node. Perhaps we thought it
self-evident in Swift.
Related RH bz#: 1797814
Change-Id: I337f8d0f1fbeee7ae927a581eecbbbcc6dc69340

Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems
Update Sphinx version as well.
Disable openstackdocs_auto_name to use 'project' variable as name.
Set openstackdocs_pdf_link to link to PDF file. Note that
the link to the published document only works on docs.openstack.org
where the PDF file is placed in the top-level html directory. The
site-preview places the PDF in a pdf directory.
Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.
Remove docs requirements from lower-constraints, they are not needed
during install or test but only for docs building.
openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.
See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
Change-Id: I131850d2a5c6164dfd48c9c95885d4754b5236c6

The requirements repo is support python 3.5 as oldest python version
while swift still supports py27.
thus, requirements-check will fail on a couple of lines in swift. The
check is only run when these files are touched.
The py2.7 packagers we know about aren't depending on upstream
requirements.txt for correctness and aside from all the production
deployments running on py2.7 we only realistically support >=py3.7
There's no good reason for our requirements.txt to be "unspported" by
the openstack requirements check job. Since they only support >=py3.5
we can change our requirements.txt inline with that. This should be
fine for everything we could hope to get out of both our
requirements.txt and the check!
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ibf8000498528c401707be8b0b91b8355cd993786

Change-Id: I227e8c2defe96fb5f33054167677ae551830d9e8

This patch removed the separate s3api, staticweb functional tests
gate jobs and added them across all other functional test jobs.
Change-Id: Ie1c606132a054defc2b3cc14a66031090e7b8449

This patch standardizes the CONTRIBUTING.rst file and adds the
required doc/source/contributor/contributing.rst
Swift already had a detailed CONTIRBUTING.rst and an informative
REVIEW_GUIDELINES.rst in the root of the repo. So we are also pulling
them into the contributor documentation so they can not only be easily
found in the checked repo but in the online documentation.
Change-Id: I4c84efbe50eb25ab922c9d6b69198dae341af48b

And stop sending WSGI strings on py3.
Change-Id: I9b769e496aa7c8ed5862c2d7310f643838328084
Closes-Bug: #1853654 

Real AWS only includes an empty delimiter element when doing a
version-aware listing.
Change-Id: Id246a157c576eac93375be084ada3740f1e09793
Closes-Bug: #1853663 

* Stop specifying logbufs=8; that's the default
* Stop including nodiratime with noatime; the latter implies the former
Nothing wrong with being explicit, I suppose, but may as well keep the
mount options to what we can easily explain: we want noatime because
Swift does not use atime, so we don't want to lose any performance to
tracking atime.
Change-Id: I1e52b4368ad7eb375964eee5132bc50297536355

Update doc examples to use explicit UUID of devices, instead of using
unstable device names.
Change-Id: I3a2eb7bbe4b4091d2567897904d939df1df6b251
Closes-Bug: #1817966 

Change-Id: I48ebbb38427612a52c36078586af7b9780b5ea37

Translate AWS S3 Object Versioning API requests to native Swift Object
Versioning API, speficially:
 * bucket versioning status
 * bucket versioned objects listing params
 * object GETorHEAD & DELETE versionId
 * multi_delete versionId
Change-Id: I8296681b61996e073b3ba12ad46f99042dc15c37
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>