openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: Iffad70b2fd901b305dc66d363039b7df44d619da

Changing the recommended ports for Swift services
from ports 6000-6002 to unused ports 6200-6202;
so they do not conflict with X-Windows or other services.
Updated SAIO docs.
DocImpact
Closes-Bug: #1521339
Change-Id: Ie1c778b159792c8e259e2a54cb86051686ac9d18

This patch removes the threads_per_disk setting. It was already a deprecated
setting and by default set to 0, which effectively meant to not use a per-disk
thread pool at all. Users are encouraged to use servers_per_port instead.
DocImpact
Change-Id: Ie76be5c8a74d60a1330627caace19e06d1b9383c

Add the ability to set the fallocate_reserve value as a percentage.
This happens automatically when adding the '%' at the end of the value.
Having the ability to set a % of free space rather than a byte value is
useful especially when drive sizes are heterogenous.
The default for fallocate_reserve has been adjusted to 1%, having the
fallocate_reserve set seems sensible for all deploys and percentages are
far safer to default than byte values (across drives of any size).
Tests added for using fallocate_reserve as a percentage.
Duplicate tests for fallocate_reserve have been removed.
Docs updated to reflect the fallocate_reserve change.
Change-Id: I4aea613a708205c917e81d6b2861396655e73238

DiskFile already fills in the _ondisk_info attribute when it tries to open
a diskfile - even if the DiskFile's fileset is not valid or deleted.
During this process the rsync tempfiles would be discovered and logged,
but no-one would attempt to clean them up - even if they were really old.
Instead of logging and ignoring unexpected files when validate a DiskFile
fileset we'll add unexpected files to the unexpected key in the
_ondisk_info attribute.
With a little bit of re-organization in the auditor's object_audit method
to get things into a single return path we can add an unconditional check
for unexpected files and remove those that are "old enough".
Since the replicator will kill any rsync processes that are running longer
than the configured rsync_timeout we know that any rsync tempfiles older
than this can be deleted.
Split unlink_older_than in common.utils into two functions to allow an
explicit list of previously discovered paths to be passed in to avoid an
extra listdir. Since the getmtime handling already ignores OSError
there's less concern of race condition where a previous discovered
unexpected file is reaped by rsync while we're attempting to clean it up.
Update some doc on the new config option.
Closes-Bug: #1554005
Change-Id: Id67681cb77f605e3491b8afcb9c69d769e154283

Swift now uses SSYNC verb instead of old REPLICATION verb for ssync
protocol. This patch replaces all docs written as REPLICATION into
SSYNC and fix a few words for explanation.
Change-Id: I1253210d4f49749e7d425d6252dd262b650d9548

account-server.conf.5
105: warning: numeric expression expected (got `)')
container-server.conf.5
111: warning: numeric expression expected (got `)')
object-expirer.conf.5
79: warning: numeric expression expected (got `)')
object-server.conf.5
114: warning: numeric expression expected (got `)')
proxy-server.conf.5
121: warning: numeric expression expected (got `)')
331: warning: numeric expression expected (got `[')
1005: warning: macro `*' not defined
Change-Id: I203dcfde83035e3b1dcb91109b72b5d08bb7840e

These errors are producing lintian warnings, so fixing them
helps having less errors when checking for Debian packages.
Change-Id: Iff99a8d5f2276515f42d758d110a43cae757db28

Change-Id: I47a3127ef698b4bd1537b1562901ee9c2b5924d4

Currently, the rsync module where the replicators send data is static. It
forbids administrators to set rsync configuration based on their current
deployment or needs.
As an example, the rsyncd configuration example encourages to set a connections
limit for the modules account, container and object. It permits to protect
devices from excessives parallels connections, because it would impact
performances.
On a server with many devices, it is tempting to increase this number
proportionally, but nothing guarantees that the distribution of the connections
will be balanced. In the worst scenario, a single device can receive all the
connections, which is a severe impact on performances.
This commit adds a new option named 'rsync_module' to the *-replicator sections
of the *-server configuration file. This configuration variable can be
extrapolated with device attributes like ip, port, device, zone, ... by using
the format {NAME}. eg:
 rsync_module = {replication_ip}::object_{device}
With this configuration, an administrators can solve the problem of connections
distribution by creating one module per device in rsyncd configuration.
The default values are backward compatible:
 {replication_ip}::account
 {replication_ip}::container
 {replication_ip}::object
Option vm_test_mode is deprecated by this commit, but backward compatibility is
maintained. The option is only effective when rsync_module is not set. In that
case, {replication_port} is appended to the default value of rsync_module.
Change-Id: Iad91df50dadbe96c921181797799b4444323ce2e

Change-Id: Ib17fc9a027db267350abc5a7db4e3172dfff7913

The actual server-side changes are simple. The tests are a different
matter. Many changes were needed to the object server tests to
handle the now-async calls to the container server. In an effort to
test this properly, some drive-by changes were made to improve tests.
I tested this patch by doing zero-byte object writes to one container
as fast as possible. Then I did it again while also saturating 2 of the
container replica's disks. The results are linked below.
https://gist.github.com/notmyname/2bb85acfd8fbc7fc312a
DocImpact
Change-Id: I737bd0af3f124a4ce3e0862a155e97c1f0ac3e52

The deprecated directive `run_pause` should be replaced with the more
standard one `interval`. The `run_pause` should be still supported for
backward compatibility. This patch updates object replicator to use
`interval` and support `run_pause`. It also updates its sample config
and documentation.
Co-Authored-By: Joanna H. Huang <joanna.huitzu.huang@gmail.com>
Co-Authored-By: Kamil Rykowski <kamil.rykowski@intel.com>
Change-Id: Ie2a3414a96a94efb9273ff53a80b9d90c74fff09
Closes-Bug: #1364735 

Fix misspellings detected by:
* pip install misspellings
* git ls-files | grep -v locale | misspellings -f -
Change-Id: I6594fc4ca5ae10bd30eac8a2f2493a376adcadee
Closes-Bug: #1257295 

Used groff to recreate the errors. I believe all the issues
except `binary-without-manpage` are solved. Would like
confirmation from someone using Lintian.
Closes-Bug: #1210114
Change-Id: I533205c53efdb7cdf3645cc3e3dc487f9ee5640a

Change the default value of wsgi workers from 1 to auto. The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.
This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.
Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident. The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.
Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.
 * drive by s/container/proxy/ in proxy-server.conf.5
 * fix misplaced max_clients in *-server.conf-sample
 * update doc/development_saio to force workers = 1
DocImpact
Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95

Change-Id: I8789d6e4d22de83db9a2760d51a94eb56a48c3b5

The new max_clients parameter allows one full control over the maximum
number of client requests that will be handled by a given worker for
any of the proxy, account, container or object servers.
Lowering the number of clients handled per worker, and raising the
number of workers can lessen the impact that a CPU intensive, or
blocking, request can have on other requests served by the same
worker.
If the maximum number of clients is set to one, then a given worker
will not perform another accept(2) call while processing, allowing
other workers a chance to process it.
DocImpact
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Change-Id: Ic01430f7a6c5ff48d7aa349dc86a5f8ac463a420

Replaced GA code for cross-domain tracking.
Patchset addresses reviewer's comments
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright
Adds current year to each Sphinx-built page.
Addresses only the docs copyright attribution, not code files.
Change-Id: Ib90fd1c92c8fafce2db821bc2b17cef1377cfc1e

A deployer may want to remove a Swift node from a load balancer for
maintenance or upgrade. This patch provides an optional mechanism for
this. The healthcheck filter config can specify "disable_path" which is
a filesystem path. If a file is present at that location, the
healthcheck middleware returns a 503 with a body of "DISABLED BY FILE".
So a deployer can configure "disable_path" and then touch that
filesystem path, wait for the proxy to be removed from the load balancer
pool, perform maintenance/upgrade, and then remove the "disable_path"
file.
Also cleaned up the conf file man pages a bit.
Change-Id: I1759c78c74910a54c720f298d4d8e6fa57a4dab4

* Add new configuration option log_address.
Change-Id: I636bd4116687629c997b70a0d804b7ed4bc46032

Change-Id: Id68231ced070b27d1f7fb5a230d345694632e7dd

"Also adding the new swift-recon and swift-ring-builder manpages to this set"
"Adding new manpages for configuration files and also making changes according to previous review suggestions"
"removing the Author line from the manpages according to suggestions"
Change-Id: I256d2b2851b55a379b59011894f214bf55ba7da9