openstack/swift - swift - OpenDev: Free Software Needs Free Tools

A container is typically sharded when it has grown to have an object
count of shard_container_threshold + N, where N <<
shard_container_threshold. If sharded using the default
rows_per_shard of shard_container_threshold / 2 then this would
previously result in 3 shards: the tail shard would typically be
small, having only N rows. This behaviour caused more shards to be
generated than desirable.
This patch adds a minimum-shard-size option to
swift-manage-shard-ranges, and a corresponding option in the sharder
config, which can be used to avoid small tail shards. If set to
greater than one then the final shard range may be extended to more
than rows_per_shard in order to avoid a further shard range with less
than minimum-shard-size rows. In the example given, if
minimum-shard-size is set to M > N then the container would shard into
two shards having rows_per_shard rows and rows_per_shard + N
respectively.
The default value for minimum-shard-size is rows_per_shard // 5. If
all options have their default values this results in
minimum-shard-size being 100000.
Closes-Bug: #1928370
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: I3baa278c6eaf488e3f390a936eebbec13f2c3e55

Make rows_per_shard an option that can be configured
in the [container-sharder] section of a config file.
For auto-sharding, this option was previously hard-coded to
shard_container_threshold // 2.
The swift-manage-shard-ranges command line tool already supported
rows_per_shard on the command line and will now also load it from a
config file if specified. Any value given on the command line takes
precedence over any value found in a config file.
Change-Id: I820e133a4e24400ed1e6a87ebf357f7dac463e38

Previously the reconstructor would quarantine isolated durable
fragments that were more than reclaim_age old. This patch adds a
quarantine_age option for the reconstructor which defaults to
reclaim_age but can be used to configure the age that a fragment must
reach before quarantining.
Change-Id: I867f3ea0cf60620c576da0c1f2c65cec2cf19aa0

See http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022718.html
and http://lists.opendev.org/pipermail/service-discuss/2021-May/000249.html
for more information.
Change-Id: I6ac37c23153c5c9d42d96e448e0656dbf8257991

Add two new sharder config options for configuring shrinking
behaviour:
 - shrink_threshold: the size below which a shard may shrink
 - expansion_limit: the maximum size to which an acceptor shard
 may grow
The new options match the 'swift-manage-shard-ranges' command line
options and take absolute values.
The new options provide alternatives to the current equivalent options
'shard_shrink_point' and 'shard_shrink_merge_point', which are
expressed as percentages of 'shard_container_threshold'.
'shard_shrink_point' and 'shard_shrink_merge_point' are deprecated and
will be overridden by the new options if the new options are
explicitly set in a config file.
The default values of the new options are the same as the values that
would result from the default 'shard_container_threshold',
'shard_shrink_point' and 'shard_shrink_merge_point' i.e.:
 - shrink_threshold: 100000
 - expansion_limit: 750000
Change-Id: I087eac961c1eab53540fe56be4881e01ded1f60e

Change the swift-manage-shard-ranges default expansion-limit to equal
the sharder daemon default merge_size i.e 750000. The previous default
of 500000 had erroneously differed from the sharder default value.
Introduce a ContainerSharderConf class to encapsulate loading of
sharder conf and the definition of defaults. ContainerSharder inherits
this and swift-manage-shard-ranges instantiates it.
Rename ContainerSharder member vars to match the equivalent vars and
cli options in manage_shard_ranges:
 shrink_size -> shrink_threshold
 merge_size -> expansion_limit
 split_size -> rows_per_shard
(This direction of renaming is chosen so that the manage_shard_ranges
cli options are not changed.)
Rename ContainerSharder member vars to match the conf file option name:
 scanner_batch_size -> shard_scanner_batch_size
Remove some ContainerSharder member vars that were not used outside of
the __init__ method:
 shrink_merge_point
 shard_shrink_point
Change-Id: I8a58a82c08ac3abaddb43c11d26fda9fb45fe6c1

I think this was mostly an artifact of the old admin-guide getting
pushed in-tree.
Change-Id: I61e63bd66ffd8207599b61721e7f0e4dc45d2e01

There are times when it is convenient to specify a policy by name or
by index (see Related-Change), but policy names can unfortunately
collide with indexes. Using a number as a policy name should at least
be discouraged.
Change-Id: I0cdd3b86b527d6656b7fb50c699e3c0cc566e732
Related-Change: Icf1517bd930c74e9552b88250a7b4019e0ab413e

Change-Id: I799bdacb7ef5850096c2178bfb12959f4205920d

Unfortunately, we can't identify the user, so we can't map to an
account, so we can't respect whatever CORS metadata might be set on the
container.
As a result, the allowed origins must be configured cluster-wide. Add a
new config option, cors_preflight_allow_origin, for that; default it
to blank (ie, deny preflights from all origins, preserving existing
behavior), but allow either a comma-separated list of origins or
* (to allow all origins).
Change-Id: I985143bf03125a05792e79bc5e5f83722d6431b3
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>

When I tried to follow documentation, the expirer said:
 object-expirer[391675]: This node is not configured to dequeue
 tasks from the legacy queue.
Our documentation was incorrect, the actual name of the option is
"dequeue_from_legacy".
Change-Id: I5ca7ac589a405d0b6250922aa9bcaabecb3c4fb0

While working on the shrinking recon drops, we want to display numbers
that directly relate to how tool should behave. But currently all
options of the s-m-s-r tool is driven by cli options.
This creates a disconnect, defining what should be used in the sharder
and in the tool via options are bound for failure. It would be much
better to be able to define the required default options for your
environment in one place that both the sharder and tool could use.
This patch does some refactoring and adding max_shrinking and
max_expanding options to the sharding config. As well as adds a
--config option to the tool.
The --config option expects a config with at '[container-sharder]'
section. It only supports the shard options:
 - max_shrinking
 - max_expanding
 - shard_container_threshold
 - shard_shrink_point
 - shard_merge_point
The latter 2 are used to generate the s-m-s-r's:
 - shrink_threshold
 - expansion_limit
 - rows_per_shard
Use of cli arguments take precedence over that of the config.
Change-Id: I4d0147ce284a1a318b3cd88975e060956d186aec

Change-Id: I33c0168c1bb89f780be6fc317a4df89322cbc28d

Previously, we would rely on replication/reconstruction to rehash
once the part power increase was complete. This would lead to large
I/O spikes if operators didn't proactively tune down replication.
Now, do the rehashing in the relinker as it completes work. Operators
should already be mindful of the relinker's I/O usage and are likely
limiting it through some combination of cgroups, ionice, and/or
--files-per-second.
Also clean up empty partitions as we clean up. Operators with metrics on
cluster-wide primary/handoff partition counts can use them to monitor
relinking/cleanup progress:
 P 3N .----------.
 a / '.
 r / '.
 t 2N / .-----------------
 i / |
 t / |
 i N -----'---------'
 o
 n
 s 0 ----------------------------------
 t0 t1 t2 t3 t4 t5
At t0, prior to relinking, there are
 N := <replica count> * 2 ** <old part power>
primary partitions throughout the cluster and a negligible number of
handoffs.
At t1, relinking begins. In any non-trivial, low-replica-count cluster,
the probability of a new-part-power partition being assigned to the same
device as its old-part-power equivalent is low, so handoffs grow while
primaries remain constant.
At t2, relinking is complete. The total number of partitions is now 3N
(N primaries + 2N handoffs).
At t3, the ring with increased part power is distributed. The notion of
what's a handoff and what's a primary inverts.
At t4, cleanup begins. The "handoffs" are cleaned up as hard links and
now-empty partitions are removed.
At t5, cleanup is complete and there are now 2N total partitions.
Change-Id: Ib5bf426cf38559091917f2d25f4f60183cd16354

Swap out the standard logger stuff in place of --logfile. Keep --device
as a CLI-only option. Everything else is pretty standard stuff that
ought to be in [DEFAULT].
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I32f979f068592eaac39dcc6807b3114caeaaa814

Change-Id: I561ca419104bfb4b64affc477aac53a2fb98137b

UpgradeImpact
=============
Operators should verify that encryption is not enabled in their
reconciler pipelines; having it enabled there may harm data durability.
For more information, see https://launchpad.net/bugs/1910804
Change-Id: I1a1d78ed91d940ef0b4eba186dcafd714b4fb808
Closes-Bug: #1910804 

When you start getting more then 3k shards in a root container the
cached shard range listing can get bigger then the default max size for
memcache (1MB).
So add a mention about it in the configuration guide.
Fixes bug 1890643
Change-Id: If380410c17ed9ebc014b8198af0ea8d502deacc8

Also fix link to python paste.
Closes-Bug: #1734740
Closes-Bug: #1719887
Change-Id: I27ff441ed746f2919bcf9ca9a77c26371b18540b

We have a separate doc page for middlewares that pulls the docstring
from each middleware's docstring[0]. This makes it easy to look up the docs
in our documentation and easy to find the middleware doc by looking in
the code of the middleware itself.
This patch does the same with the audit watchers. There is now a page
that generates a list of audit watchers, even though currently it's only
one, and pulls the docs from their docstrings. Giving us an easy way to
maintain each audit watcher doc along with it's code.
[0] - https://docs.openstack.org/swift/latest/middleware.html
Change-Id: I1456aba0158d29fa0a879dcc2dfb13245c45ad16

Swift operators may find it useful to operate on each object in their
cluster in some way. This commit provides them a way to hook into the
object auditor with a simple, clearly-defined boundary so that they
can iterate over their objects without additional disk IO.
For example, a cluster operator may want to ensure a semantic
consistency with all SLO segments accounted in their manifests,
or locate objects that aren't in container listings. Now that Swift
has encryption support, this could be used to locate unencrypted
objects. The list goes on.
This commit makes the auditor locate, via entry points, the watchers
named in its config file.
A watcher is a class with at least these four methods:
 __init__(self, conf, logger, **kwargs)
 start(self, audit_type, **kwargs)
 see_object(self, object_metadata, data_file_path, **kwargs)
 end(self, **kwargs)
The auditor will call watcher.start(audit_type) at the start of an
audit pass, watcher.see_object(...) for each object audited, and
watcher.end() at the end of an audit pass. All method arguments are
passed as keyword args.
This version of the API is implemented on the context of the
auditor itself, without spawning any additional processes.
If the plugins are not working well -- hang, crash, or leak --
it's easier to debug them when there's no additional complication
of processes that run by themselves.
In addition, we include a reference implementation of plugin for
the watcher API, as a help to plugin writers.
Change-Id: I1be1faec53b2cdfaabf927598f1460e23c206b0a

Add a new config option to SLO, allow_async_delete, to allow operators
to opt-in to this new behavior. If their expirer queues get out of hand,
they can always turn it back off.
If the option is disabled, handle the delete inline; this matches the
behavior of old Swift.
Only allow an async delete if all segments are in the same container and
none are nested SLOs, that way we only have two auth checks to make.
Have s3api try to use this new mode if the data seems to have been
uploaded via S3 (since it should be safe to assume that the above
criteria are met).
Drive-by: Allow the expirer queue and swift-container-deleter to use
high-precision timestamps.
Change-Id: I0bbe1ccd06776ef3e23438b40d8fb9a7c2de8921

This patch moves the tables describing configuration options for each
server type from the deployment_guide.rst doc to separate per-server
documents. The new per-server documents are grouped under a config
directory with a config index doc. The config index doc is listed in
the top level index and provides a single starting point to navigate
to the individual server docs.
Change-Id: I6cedd98586febb5dc949c088ee44e160385ed324

We sometimes see probe test failures in the gate where `resetswift`
fails like
 umount: /mnt/sdb1: target is busy.
It seems to be because `swift-init all kill` is just a stop with a
status check after the fact. Add an orphan sweep afterwards to make
sure worker subprocesses got cleaned up, too.
Change-Id: I533b436a35e5c3deb7488e9961b55f2274c14751

Capture the on the wire status code for logging because we change the
logged status code sometimes.
Closes-Bug: #1896518
Change-Id: I27feabe923a6520e983637a9c68a19ec7174a0df

docs: Removing the use of NameVirtualHost from the apache examples
It's not used anymore. It's deprecated in fact: https://httpd.apache.org/docs/2.4/mod/core.html#namevirtualhost
Change-Id: I76999cfacc10a244024ee0cca66dda95a0169a67
docs: Added more spacing to the apache2 examples
They're easier to read and a bit less bloated.
Change-Id: I5e21a66018b7ef309918fbbde93f2494286d291e
docs: Switching to /srv/www to be more FHS 3.0 conformat
It's more modern and well supported to use /srv/www now in place of
/var/www.
Change-Id: Icd09ed4d5fb4e2b9b84ddead21313ea1c0a87c91
ref: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s17.html
docs: Added user, group and display name in WSGI examples
This properly sets the user and group for the wsgi processes in the
examples; as well as adding a display name for easier identification.
Change-Id: Ie5783081e4054e5b2fbf3a856716101a1aaf61b8
docs: Replace apachectl for systemctl commands
It's safe to asume that all modern distros; supported by OpenStack, will
have systemd implemented. It's better to favor systemctl in those cases.
Change-Id: Ic0d2e47c1ac53502ce638d6fc2424ab9df037262
docs: Emphasis to file paths and command options
I've enclosed configuration options or parameters in interpreted text
quotes.
Also, I've enclosed fiel paths with inline literal quotes.
Change-Id: Iec54b7758bce01fc8e8daff48498383cb70c62ce
docs: Fixed wording used to indicate the restart of apache
Just a little commit to make it clearer of what we're gonna do.
Change-Id: Id5ab3e94519bcfe1832b92e456a1d1fa81dd54e3

It's more modern and well supported to use /srv/www now in place of
/var/www.
Change-Id: Icd09ed4d5fb4e2b9b84ddead21313ea1c0a87c91
ref: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s17.html 

Change-Id: Ic07b0bdd5f1c90e7e25991def58c587680d7bfab

It does *not* include transfer time; it's just the time until response
is started.
Change-Id: I67c5c37c8c669e1e2de2cbfca8bcec02dc98659f

Note that existing SAIOs with 60xx ports should still work fine.
Change-Id: If5dd79f926fa51a58b3a732b212b484a7e9f00db
Related-Change: Ie1c778b159792c8e259e2a54cb86051686ac9d18

Lower the part-power -- 18 is way higher than is needed for a dev
environment.
Add commands for reduced-redundancy and erasure-coded storage policies.
Related-Change: Ibe46011d8e6a6482d39b3a20ac9c091d9fbc6ef7
Related-Change: I6f11f7a1bdaa6f3defb3baa56a820050e5f727f1
Related-Change: I0403016a4bb7dad9535891632753b0e5e9d402eb
Change-Id: I13de27674c81977c2470d43bbb2126ecc4bdd85a

My customers use the upstream documentation for manual
installation of Swift nodes and they come upon the lack
of any mention that host firewall ought to accomodate
services running on the node. Perhaps we thought it
self-evident in Swift.
Related RH bz#: 1797814
Change-Id: I337f8d0f1fbeee7ae927a581eecbbbcc6dc69340

Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems
Update Sphinx version as well.
Disable openstackdocs_auto_name to use 'project' variable as name.
Set openstackdocs_pdf_link to link to PDF file. Note that
the link to the published document only works on docs.openstack.org
where the PDF file is placed in the top-level html directory. The
site-preview places the PDF in a pdf directory.
Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.
Remove docs requirements from lower-constraints, they are not needed
during install or test but only for docs building.
openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.
See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
Change-Id: I131850d2a5c6164dfd48c9c95885d4754b5236c6

The requirements repo is support python 3.5 as oldest python version
while swift still supports py27.
thus, requirements-check will fail on a couple of lines in swift. The
check is only run when these files are touched.
The py2.7 packagers we know about aren't depending on upstream
requirements.txt for correctness and aside from all the production
deployments running on py2.7 we only realistically support >=py3.7
There's no good reason for our requirements.txt to be "unspported" by
the openstack requirements check job. Since they only support >=py3.5
we can change our requirements.txt inline with that. This should be
fine for everything we could hope to get out of both our
requirements.txt and the check!
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ibf8000498528c401707be8b0b91b8355cd993786

Change-Id: I227e8c2defe96fb5f33054167677ae551830d9e8

This patch removed the separate s3api, staticweb functional tests
gate jobs and added them across all other functional test jobs.
Change-Id: Ie1c606132a054defc2b3cc14a66031090e7b8449