openstack/swift - swift - OpenDev: Free Software Needs Free Tools

- Add proxy-logging to multinode. We had it since Folsom and people
 still forget it, resulting in missing logs.
- Use correct name, for ease hit with '*' in vi at least.
Admittedly trivial changes, which I meant to hold until Leah's major
doc improvement lands, but I'm tired of keeping stuff like this in
my working repo.
Change-Id: I44f80c51d6d7329a9b696e67fcb8a895db63e497

Change-Id: I8789d6e4d22de83db9a2760d51a94eb56a48c3b5

DocImpact
If account reaper has not managed to clean out an account after a long
period, it prints a message to the log (you can search your system looking
for such messages). Introduce reap_warn_after config variable to determine
when to emit the message (defaults to 30 days).
Also fix bug 1181995 (edge case where object name is an empty string)
Change-Id: Ic0dfee04742d06b6a51b59f302d7a272d7c1de92

The crossdomain doc was named *.xml instead of *.rst causing it to not
get built or included in the toctree where it was supposed to.
The apache deployment guide wasn't linked to from anywhere, so I added
it under the normal deployment guide.
Change-Id: I817a1f2ca1ed7913e8ea5155cc1fac07caf0b637

Allow Swift daemons and servers to optionally accept a directory as the
configuration parameter. Directory based configuration leverages
ConfigParser's native multi-file support. Files ending in '.conf' in the
given directory are parsed in lexicographical order. Filenames starting with
'.' are ignored. A mixture of file and directory configuration paths is not
supported - if the configuration path is a file behavior is unchanged.
 * update swift-init to search for conf.d paths when building servers
 (e.g. /etc/swift/proxy-server.conf.d/)
 * new script swift-config can be used to inspect the cumulative configuration
 * pull a little bit of code out of run_wsgi and test separately
 * fix example config bug for the proxy servers client_disconnect option
 * added section on directory based configuration to deployment guide
DocImpact
Implements: blueprint confd
Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b

The new max_clients parameter allows one full control over the maximum
number of client requests that will be handled by a given worker for
any of the proxy, account, container or object servers.
Lowering the number of clients handled per worker, and raising the
number of workers can lessen the impact that a CPU intensive, or
blocking, request can have on other requests served by the same
worker.
If the maximum number of clients is set to one, then a given worker
will not perform another accept(2) call while processing, allowing
other workers a chance to process it.
DocImpact
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Change-Id: Ic01430f7a6c5ff48d7aa349dc86a5f8ac463a420

Allows client-side technologies such as Flash, Java and Silverlight running
on web pages served elsewhere to interact with the Swift API.
Bug #1159960
Change-Id: I7d0533a0aaf189ac452abbd983469acb064fdca4

The lack of any documentation for the auth_version was observed
by Red Hat QA, see rhbz#955247.
Change-Id: I0afc04d848082a80480724bdbf99031e4b99e38e

Support separate replication ip address:
- Added new function in utils. This function provides ability
 to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
 Replication process uses new function now.
Replication network parameters:
- Replication network fields (replication_ip, replication_port)
 support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
 functions.
Implementation of replication servers:
- Separate replication servers use the same code as normal replication
 servers, but with replication_server parameter = True. When using a
 separate replication network, the non-replication servers set
 replication_server = False. When there is no separate replication
 network (the default case), replication_server is not included in the config.
DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network

dnspython was added recently to the list of requirments.
This patch completes the change:
1. The new requirement is documented
2. The need for tox --recreate is documented
(assuming there is no way to automate 2 above)
Change-Id: I49f0cb4b01b79dd7cf4329984a2f8416378ebe19
Fixes: Bug #1176627 

Change-Id: I49f8c272457547d8aae853986573549828d2503a

The trans_id_suffix (default is empty) would be appended to the swift transaction
id allowing one to easily figure out from which cluster that X-Trans-Id
belongs to. This is very useful when one is managing more than one swift
cluster. Also updated sample and manpage to reflect the changes.
Change-Id: Icdf63643e9c1bde36a9ef5e3f41ee9fb20e55f5d

Extensive refactor here to consolidate what nodes are contacted for
any request. This consolidation means reads will contact the same set
of nodes that writes would, giving a very good chance that
read-your-write behavior will succeed. This also means that writes
will not necessarily try all nodes in the cluster as it would
previously, which really wasn't desirable anyway. (If you really want
that, you can set request_node_count to a really big number, but
understand that also means reads will contact every node looking for
something that might not exist.)
* Added a request_node_count proxy-server conf value that allows
 control of how many nodes are contacted for a normal request.
In proxy.controllers.base.Controller:
* Got rid of error_increment since it was only used in one spot by
 another method and just served to confuse.
* Made error_occurred also log the device name.
* Made error_limit require an error message and also documented a bit
 better.
* Changed iter_nodes to just take a ring and a partition and yield
 all the nodes itself so it could control the number of nodes used
 in a given request. Also happens to consolidate where sort_nodes is
 called.
* Updated account_info and container_info to use all nodes from
 iter_nodes and to call error_occurred appropriately.
* Updated GETorHEAD_base to not track attempts on its own and just
 stop when iter_nodes tells it to stop. Also, it doesn't take the
 nodes to contact anymore; instead it takes the ring and gets the
 nodes from iter_nodes itself.
Elsewhere:
* Ring now has a get_part method.
* Made changes to reflect all of the above.
Change-Id: I37f76c99286b6456311abf25167cd0485bfcafac

- We allow all headers requested in preflight request. The CORS
 specification does leave the door open for this, as mentioned in
 http://www.w3.org/TR/cors/#resource-preflight-requests
 Note: Since the list of headers can be unbounded
 simply returning headers can be enough.
- This is a followup to review:
 https://review.openstack.org/#/c/24415/.
- Fixes bug 1155034.
Change-Id: If7b8f2f3a581c5209892d1ccc9f06ddb8fac92dd

For multiple node swift cluster setup, these directories should be
created in the setup steps. Otherwise, when start up storage servers
there will be errors. The error messages won't tell if it is because
of missing the directory.
Change-Id: I67d74abb7743b76739b5e747d6dcbd3214b00774
Fixes: bug #1158519 

Change-Id: I3dc7930ee2a1eb26b1f60e83fce2bc1bea0a8b0f

A new configuration parameter is added to /etc/swift/swift.conf
[swift-hash]
swift_hash_path_prefix = 'random unique string'
New installations are advised to set this parameter to a random secret,
which would not be disclosed ouside the organization.
The same secret needs to be used by all swift servers of the same cluster.
Existing installations should set this parameter to an empty string
(the default)
DocImpact
Fixes: Bug #1157454
Change-Id: I63b10d0b7d6dd3f74e0f10bb41b5f240fa03578a

Instructions for a Multiple Server Swift Installation (Ubuntu) doc
for creating /var/run/swift and changing owner command were not
using the right format. Missing two colons.
Change-Id: Ie23007a0da498373fbfb137c7edb3d80813c6ba5
Fixes: bug #1158310 

Multiple Server Swift Installation (Ubuntu) instruction does not
indicate that the directory /var/run/swift needs to be created.
That directory actually needs to be created and the ownship needs
to be changed to the user/group which swift service runs under.
This patch will fix the document and gives the steps how to create
the directory and set the ownership right. It also gives instruction
on how the script can be added so that swift services can be resarted
after system reboots.
Change-Id: Id61aa67cc0d6f66d749701e6ea824b1ff3b2c478
Fixes: bug #1156631 

Apparently the use of port 6030 was causing some confusion.
Fixes bug 1095474.
Change-Id: I0cc71f4733ad91694e015a9b75c3eda080aca6fb

We can set x-versions-location empty to remove this header in API, but
to keep consistent, we should allow x-remove-versions-location too. The
usage is post "x-remove-versions-locaion: x", just like ACL remove
headers.
Fixed: bug #1107592
Change-Id: I1271eec6401d4a0e8c1a7c2d63aeb8dfef00bf6d

The region is one level above the zone; it is intended to represent a
chunk of machines that is distant from others with respect to
bandwidth and latency.
Old rings will default to having all their devices in region 1. Since
everything is in the same region by default, the ring builder will
simply distribute across zones as it did before, so your partition
assignment won't move because of this change. If you start adding
devices in other regions, of course, the assignment will change to
take that into account.
swift-ring-builder still accepts the same syntax as before, but will
default added devices to region 1 if no region is specified.
Examples:
$ swift-ring-builder foo.builder add r2z1-1.2.3.4:555/sda
$ swift-ring-builder foo.builder add r1z3-1.2.3.4:555/sda
$ swift-ring-builder foo.builder add z3-1.2.3.4:555/sda
Also, some updates to ring-overview doc.
Change-Id: Ifefbb839cdcf033e6c9201fadca95224c7303a29

The recent account_quotas (https://review.openstack.org/23434)
patch added a new setting request.environ[reseller_request].
This patch adds tests for tempauth and keystoneauth as well as
an updated overview_auth.rst.
Change-Id: Icdb7ec9948ae7424b0721fc51a143782b2fdc5a6

container-sync now skips faulty objects in the first and second rounds.
All replicas try in the second round.
No server will give up until the faulty object suceeds
Fixes: bug #1068423
Change-Id: I0defc174b2ce3796a6acf410a2d2eae138e8193d

Add a new middleware implementing account quotas.
This middleware blocks write requests (PUT, POST) if a given quota (in bytes)
is exceeded while DELETE requests are still allowed.
Quotas are stored in the x-account-meta-quota-bytes metadata entry.
Write requests to this metadata setting are only allowed for resellers.
Change-Id: I57fd7c6209f34cc79d4bab72d500d43ba2a62083

Add support for functional tests that work with Apache web front end
Change-Id: I72358a12016eeccc842d834461dbebaa188aa117
Implements: blueprint wsgi-application-interface

Implements blueprint list-endpoints.
DocImpact: new middleware list_endpoints.
Change-Id: I0c4911ff726abd4cb8ce2b6245c99786ad46b410

DocImpact
Change-Id: I7edaa5e44208ab451f7f7566b64bb571b8eea1f9

Fixes bug 1104708
There could be severe performance drop for swift is one disk of one
storage node is problematic due to the tragic state of async disk I/O.
This patch provided PUT timing per kB transfered (ms/kB) monitoring
support for each non-zero-byte request of each disk and report to
statsD for alert.
-adding "object-server.PUT.<device>.timing" metrics for object-server.
DocImpact.
Change-Id: Ie94bddad28e8be52e71683bf6c9db988664abe47

When someone split away python-swiftclient, he left behind the
client manpage.
Apparently all we need now is to delete it from swift and
add it to python-swiftclient. Except that of course the releases
have to be synchronized somehow or else files conflict.
Change-Id: I8b5f7b5557c28818048d8941df36473dacfb1d57

- Things swill go badly with swift if we leave the default to authtoken
 to use its own memcache cache connection based python-memcache c based
 binding.
Change-Id: I293b875acdcb06e5a7a0cfa9a9bb5d7678675da0

Fix bug 1129760
Without speed limit, DB auditor will likely consume high CPU% on
storage node. That will highly impact the cluster's performance.
This patch adds two options for account/container auditor:
 - containers_per_second: Maximum containers audited per second
 - accounts_per_second: Maximum accounts audited per second
DocImpact
Change-Id: I9faa506438185a83ca77db4906969328624d015f