openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Replacing six.iter usages substituting .items(), and
.values() in place of six.iteritems/dict.iteritems,
and six.itervalues/itervalues respectively.
1.As mentioned in [1], we should avoid using six.iteritems
to achieve iterators. We can use dict.items instead, as it
will return iterators in PY3 as well. And dict.items/keys
will more readable.
2.In py2, the performance about listshould be negligible,
see the link [2].
[1] https://wiki.openstack.org/wiki/Python3#Common patterns
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: I53e3ed77c8279ac48f738e396672794300de440d

Fix typos that found in swift.
Change-Id: I52fad1a4882cec4456f22174b46d54e42ec66d97

This patch adds support for retrieving the encryption root secret from
an external key management system. In practice, this is currently
limited to Barbican.
Change-Id: I1700e997f4ae6fa1a7e68be6b97539a24046e80b

* Update URLs according to document migration
 * Update the dead and outdated links
Change-Id: Id92552f4a2d0fb79ddefc55a08636f2e7aeb07cb

* Fixes warnings in RST file
* Suppress warning log from pyeclib during the doc build.
 pyeclib emits a warning message on an older liberasurecode [1]
 and sphinx treats this as error (when warning-is-error is set).
 There is no need to check warnings during the doc build,
 so we can safely suppress the warning.
This is a part of the doc migration community-wide effort.
http://specs.openstack.org/openstack/docs-specs/specs/pike/os-manuals-migration.html
[1] https://github.com/openstack/pyeclib/commit/d163972b
Change-Id: I9adaee29185a2990cc3985bbe0dd366e22f4f1a2

also added a test redirect so we know if things are working
Change-Id: I8fd848c23053e52e45fa55637f280124eba74927

This change adds a new Strategy concept to the daemon module similar to
how we manage WSGI workers. We need to leverage multiple python
processes to get the concurrency properties we need. More workers will
rebalance much faster on dense chassis with many devices.
Currently the default is still only one process, and no workers. Set
reconstructor_workers in the [object-reconstructor] section to some
whole number <= the number of devices on a node to get that many
reconstructor workers.
Each worker will operate on a different subset of disks.
Once mode works as before, but tends to want to update recon drops a
little bit more.
If you change the rings, the strategy will shutdown workers and spawn
new ones.
You can kill the worker pids and the daemon strategy will respawn them.
New per-disk reconstructor stats are dumped to recon under the
object_reconstruction_per_disk key. To maintain legacy compatibility
and replication monitoring based on cycle times they are aggregated
every stats_interval (default 5 mins).
Change-Id: I28925a37f3985c9082b5a06e76af4dc3ec813abe

Clarify in usage statement and man pages that CLI override options for
swift-object-reconstructor and swift-object-replicator only have
effect when --once is used.
Also add a link to object reconstructor source code docs to the doc
index page for consistency with the other object services.
Change-Id: If348b340d59a672d3a19d4df231ebdb74f4aed51

Previously it was hard to navigate to a particular config section in
the deployment guide, and not possible to provide a link directly to
one section.
This patch makes each config section a heading so that it appears in
navigation tables and can be easily linked to. A list of config
sections is also added at the start of each server section.
Change-Id: Iecb0637fde521600a9163fa66b3dbdc176a71dff
Related-Bug: #1626290 

Change-Id: I062998e813718828b7adf4e7c3f877b6a31633c0
Closes-Bug: #1626290 

* Change some absolute URLs to internal links
* Fix some bulletted list indentation
* Choose a better lexer for some syntax highlighting
* Use ``inline code`` instead of `italics` for some example command
 lines
* Change some quoted paragraphs that only included inlined code to be
 proper code blocks
Change-Id: Iaaa7eefb690122f5af9dcb1c871358c22335c743

Change-Id: I2f63ce16ae67277504a47e68c7dca3c3412fe92c

Change-Id: I966f849804cd5eeeae566bc7e732c7bf56bc9eaf

Change-Id: I005232d95a3e1d181271eef488a828ad330e6006

Context for this is at
https://specs.openstack.org/openstack/docs-specs/specs/pike/os-manuals-migration.html
Change-Id: I9a4da27ce1d56b6406e2db979698038488f3cf6f

As part of the docs migration work[0] for Pike we need to switch to use
the openstackdocstheme.
Fix one display problem with wrong section markup in the index file.
[0]https://review.openstack.org/#/c/472275/
Change-Id: Ide31218a7f37ba5d959de99cab48fc6513bf426f

Currently all devices in the ring and all services in a SAIO
all bind to the same loopback address 127.0.0.1. But this
breaks servers_per_port if you want to do any testing on that.
This change binds each service to a different loopback address
and updates the rings (remakerings) accordingly.
To make sure rysncd binds correctly the bind address needed
to be changed to listen on all addresses (0.0.0.0).
Change-Id: I7e77434f275df1e2699de495d8b622b90157a9d7

When deleting objects in multi-region swift delpoyment with write
affinity configured, users always get 404 when deleting object before
it's replcated to approriate nodes.
This patch adds a config item 'write_affinity_handoff_delete_count' so
that operator could define how many local handoff nodes should swift
send request to get more candidates for the final response, or by
default just leave it to swift to calculate the appropriate number.
Change-Id: Ic4ef82e4fc1a91c85bdbc6bf41705a76f16d1341
Closes-Bug: #1503161 

If you're running servers_per_port > 0 and threads_per_disk = 0 (as it
should be with servers_per_port on), each object-server process will
have 20 IO threads waiting around to service eventlet.tpool
calls. This is far too many; with servers_per_port, there's no real
benefit to having so many IO threads.
This commit makes it so that, when servers_per_port > 0, each object
server defaults to having one main thread and one IO thread.
Also, eventlet's tpool size is now configurable via the object-server
config file. If a tpool size is set, that's what we'll use regardless
of servers_per_port. This allows operators with an excess of threads
to remove some regardless of servers_per_port.
Change-Id: I8f8914b7e70f2510393eb7c5e6be9708631ac027
Closes-Bug: 1554233

container and object updaters sleeps "slowdown" (default 0.01) seconds
after every processed container/object. Because time.sleep call adds overhead,
use ratelimit_sleep from common.utils instead. Same as in auditor.
Change-Id: I362aa0f13c78ad03ce1f76ee0257b0646f981212

We said we were going to do it, we've had two releases saying we'd do
it, we've even backported our saying it to Newton -- let's actually do
it.
Upgrade Consideration
=====================
Erasure-coded storage policies using isa_l_rs_vand and nparity >= 5 must
be configured as deprecated, preventing any new containers from being
created with such a policy. This configuration is known to harm data
durability. Any data in such policies should be migrated to a new
policy. See https://bugs.launchpad.net/swift/+bug/1639691 for more
information.
UpgradeImpact
Related-Change: I50159c9d19f2385d5f60112e9aaefa1a68098313
Change-Id: I8f9de0bec01032d9d9b58848e2a76ac92e65ab09
Closes-Bug: 1639691

This patch adds methods to increase the partition power of an existing
object ring without downtime for the users using a 3-step process. Data
won't be moved to other nodes; objects using the new increased partition
power will be located on the same device and are hardlinked to avoid
data movement.
1. A new setting "next_part_power" will be added to the rings, and once
the proxy server reloaded the rings it will send this value to the
object servers on any write operation. Object servers will now create a
hard-link in the new location to the original DiskFile object. Already
existing data will be relinked using a new tool in the new locations
using hardlinks.
2. The actual partition power itself will be increased. Servers will now
use the new partition power to read from and write to. No longer
required hard links in the old object location have to be removed now by
the relinker tool; the relinker tool reads the next_part_power setting
to find object locations that need to be cleaned up.
3. The "next_part_power" flag will be removed.
This mostly implements the spec in [1]; however it's not using an
"epoch" as described there. The idea of the epoch was to store data
using different partition powers in their own namespace to avoid
conflicts with auditors and replicators as well as being able to abort
such an operation and just remove the new tree. This would require some
heavy change of the on-disk data layout, and other object-server
implementations would be required to adopt this scheme too.
Instead the object-replicator is now aware that there is a partition
power increase in progress and will skip replication of data in that
storage policy; the relinker tool should be simply run and afterwards
the partition power will be increased. This shouldn't take that much
time (it's only walking the filesystem and hardlinking); impact should
be low therefore. The relinker should be run on all storage nodes at the
same time in parallel to decrease the required time (though this is not
mandatory). Failures during relinking should not affect cluster
operations - relinking can be even aborted manually and restarted later.
Auditors are not quarantining objects written to a path with a different
partition power and therefore working as before (though they are reading
each object twice in the worst case before the no longer needed hard
links are removed).
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
[1] https://specs.openstack.org/openstack/swift-specs/specs/in_progress/
increasing_partition_power.html
Change-Id: I7d6371a04f5c1c4adbb8733a71f3c177ee5448bb

Change-Id: Ie51ea5c729341da793887e1e25c1e45301a96751

Soften the language about inefficiency on read and strengthen the
language encouraging the use of read affinity and composite rings.
Change-Id: Idc81a8c71e74ae28d384759700c5268d77ae3c85

* In light of the composite rings feature being added [1],
 downgrade the warnings about EC Duplication [2] being
 experimental.
* Add links from Global EC docs to composite rings and
 per-policy proxy config features.
* Add discussion of using EC duplication with composite
 rings.
* Update Known Issues.
[1] Related-Change: I0d8928b55020592f8e75321d1f7678688301d797
[2] Related-Change: Idd155401982a2c48110c30b480966a863f6bd305
Change-Id: Id97a4899255945a6eaeacfef12fd29a2580588df

That option was removed entirely in 2.8.0.
Change-Id: Ib40f816936429a78e622d3737bb0b064225d2d44
Related-Change: Ie76be5c8a74d60a1330627caace19e06d1b9383c

For various reasons, an operator might want to use specifics nameservers
instead of the systems ones to resolve CNAME in cname_lookup. This patch
creates a new configuration variable nameservers which accepts a list of
nameservers separated by commas. If not specified or empty, systems
namservers are used as previously.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I34219e6ab7e45678c1a80ff76a1ac0730c64ddde

The description of storage policy config options was
unstructured and repetitive. This patch attempts to
improve the doc by gathering the notes for each option
into a structured list.
Change-Id: I57090b35a70f365e82fb0e29ab42e533d6359a7b

- add proxy server per policy config as an optional
 step in the configuration of a policy, with link to
 the deployment guide
- add reverse link from deployment guide per-policy
 config doc section to storage policies docs
Drive-by fix an incorrect test comment
Change-Id: Ib95310193270a63c9d1e321c6e7de240e00b387f
Related-Change: I3f718f425f525baa80045ba067950c752bcaaefc

Instead, link to the middleware list and auth overview, as well as
referring readers to proxy-server.conf-sample
TempAuth-related content that was previously in the deployment guide has
been moved to TempAuth's own docs, which have been cleaned up a bit.
Change-Id: I00070bb09294362c069f7ee9426ac570bc1b3ddb

This is an alternative approach to that proposed in [1]
Adds support for optional per-policy config sections
to be added in proxy-server.conf. This is highly desirable
to allow per-policy affinity options to be set for use with
duplicated EC policies [2] and composite rings [3].
Certain options found in per-policy conf sections will
override their equivalents that may be set in the
[app:proxy-server] section. Currently the options
handled that way are:
 sorting_method
 read_affinity
 write_affinity
 write_affinity_node_count
For example:
 [proxy-server:policy:0]
 sorting_method = affinity
 read_affinity = r1=100
 write_affinity = r1
 write_affinity_node_count = 1 * replicas
The corresponding attributes of the proxy-server Application
are now available from instances of an OverrideConf object
that is obtained from Application.get_policy_options(policy).
[1] Related-Change: I9104fc789ba85ab3ab5ccd34096125b482821389
[2] Related-Change: Idd155401982a2c48110c30b480966a863f6bd305
[3] Related-Change: I0d8928b55020592f8e75321d1f7678688301d797
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I3f718f425f525baa80045ba067950c752bcaaefc