openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I307baf793cd39786b3a8a921f5b02c945e1dbf46

This patch removes an additional 'and' added to a sentence.
Change-Id: I6ab47637ef2d2f97f35188a84e741274c2b504e8

replace a OpenStack bucket with an OpenStack bucket
Change-Id: I4ada68720dcf23fb9b94c922ddb3eefd42aa80a7

Change-Id: I8a179e2e9dea1dac57a14696bc0abcd48753adfc

This patch adds a read_only middleware to swift. It gives the ability
to make an entire cluster or individual accounts read only.
When a cluster or an account is in read only mode, requests that would
result in writes to the cluser are not allowed.
DocImpact
Change-Id: I7e0743aecd60b171bbcefcc8b6e1f3fd4cef2478

Previously, these headers had to be added by operators to their
object-server.conf when enabling swift3 middleware. Since s3api
is now imported into swift we should go ahead and add these headers
by default too.
Change-Id: Ib82e175096716e42aecdab48f01f079e09da6a1d
Signed-off-by: Thiago da Silva <thiago@redhat.com>

Change-Id: I25bdb2020c10f4b3aede6994e9ffc3f05a054c29

Move the doc for manage-shard-ranges to the manage_shard_ranges.py module
and include it in overview_container_sharding.rst. This makes the doc for
manage-shard-ranges more obvious when viewing the code.
Change-Id: I27ca9b59897c5256dd5e2c3d4e26ff9e762b4a81

Change-Id: I1e736c8da142adecc2b0b629cfd48afa072d40a0

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I964666d2c1ce893326c6aa2bbe9e1dd0312e7a9e

Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: John Dickinson <me@not.mn>
Change-Id: I0693e54c1d7f3b77f53c3df5c616a16f74723b97

The sharder daemon visits container dbs and when necessary executes
the sharding workflow on the db.
The workflow is, in overview:
- perform an audit of the container for sharding purposes.
- move any misplaced objects that do not belong in the container
 to their correct shard.
- move shard ranges from FOUND state to CREATED state by creating
 shard containers.
- move shard ranges from CREATED to CLEAVED state by cleaving objects
 to shard dbs and replicating those dbs. By default this is done in
 batches of 2 shard ranges per visit.
Additionally, when the auto_shard option is True (NOT yet recommeneded
in production), the sharder will identify shard ranges for containers
that have exceeded the threshold for sharding, and will also manage
the sharding and shrinking of shard containers.
The manage_shard_ranges tool provides a means to manually identify
shard ranges and merge them to a container in order to trigger
sharding. This is currently the recommended way to shard a container.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7f192209d4d5580f5a0aa6838f9f04e436cf6b1f

Fix indentation of nested list.
Old indentation was wrongly interpreted as quotation
Change-Id: Ia9625bc3b7984501f4931b3428fa930a156bfe66

This patch is one of follow up to remove unnecessary
files, and a comment in the code.
The conf files are used to setup functests environment
in the past swift3 repository but that should port to
setuppers of functests (see related change). Anyway,
we don't need shell based older conf.in script on that way.
Change-Id: If431979ea6fa373ac1cde4b7e13d57d91fb15be8
Related-Change: I6f30f74678ad35479da237361bee48c46c0ecc49

This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.
1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)
2. Port unittests to test/unit/s3api dir to be able to run on the gate.
3. Port functests to test/functional/s3api and setup in-process testing
4. Port docs to doc dir, then address the namespace change.
5. Use get_logger() instead of global logger instance
6. Avoid global conf instance
Ex. fix various minor issue on those steps (e.g. packages, dependencies,
 deprecated things)
The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)
Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.
Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>

Add a multiprocess mode to the object replicator. Setting the
"replicator_workers" setting to a positive value N will result in the
replicator using up to N worker processes to perform replication
tasks.
At most one worker per disk will be spawned, so one can set
replicator_workers=99999999 to always get one worker per disk
regardless of the number of disks in each node. This is the same
behavior that the object reconstructor has.
Worker process logs will have a bit of information prepended so
operators can tell which messages came from which worker. It looks
like this:
 [worker 1/2 pid=16529] 154/154 (100.00%) partitions replicated in 1.02s (150.87/sec, 0s remaining)
The prefix is "[worker M/N pid=P] ", where M is the worker's index, N
is the total number of workers, and P is the process ID. Every message
from the replicator's logger will have the prefix; this includes
messages from down in diskfile, but does not include things printed to
stdout or stderr.
Drive-by fix: don't dump recon stats when replicating only certain
policies. When running the object replicator with replicator_workers >
0 and "--policies=X,Y,Z", the replicator would update recon stats
after running. Since it only ran on a subset of objects, it should not
update recon, much like it doesn't update recon when run with
--devices or --partitions.
Change-Id: I6802a9ad9f1f9b9dafb99d8b095af0fdbf174dc5

Pypi url changed from [1] to [2]
[1] https://pypi.python.org/pypi/<package>
[2] https://pypi.org/project/<package>
Change-Id: Ic639298e83189410395f31cc9ef60076af479efd

Change-Id: I04f5c14586ad4ecac03d18992b8cf351d6eab1e3

Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.
[1]https://review.openstack.org/#/c/508522/
Change-Id: I43bbc8b8c986e54a9a0829a0631d78d4077306f8

* Cleaned up the SDK/library links
* Added a few projects
* Fixed some existing links
* Removed some very old, unmaintained projects
Change-Id: I3effd920e978eb7af39ab27b4877a7bfc8c64b8b

For compliance with the Project Testing Interface as described in:
https://governance.openstack.org/tc/reference/project-testing-interface.html
For more details information, please refer to:
http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html
Co-Authored-By: Nguyen Hai <nguyentrihai93@gmail.com>
Change-Id: I26dc41c7df57bf79db531c6e67e148e01c17e992

Change-Id: Id6678dd7e59a85bc2a7b3ba3d41d28e08956ae3e

TrivialFix
[1] is the installation guide for OpenStack components, obviously,
we need [1] in the docs.
[1] https://docs.openstack.org/latest/install/
Change-Id: I3c6fe7327f5552cc2b8f0f0e42b41f8e989a0a7e

Change-Id: I7b2dd19f18cbfac3bc3ea763cd3333e3fb68f1cd

Curly quotes(Chinese punctuation) usually input from Chinese input method.
When read from english context, it makes some confusion.
Change-Id: Ibd50299ee287c56ec4759ea8ff53d47d006144f8

Change-Id: I8d13696970593d47d0f07cf97aa1284eed5f7a15

Change-Id: I9a04cb2fbe61e1fbd8185ab2fac9abbcea4d55cc

The object updater has five different stats, but its logging only told
you two of them (successes and failures), and it only told you after
finishing all the async_pendings for a device. If you have a cluster
that's been sick and has millions upon millions of async_pendings
laying around, then your object-updaters are frustratingly
silent. I've seen one cluster with around 8 million async_pendings per
disk where the object-updaters only emitted stats every 12 hours.
Yes, if you have StatsD logging set up properly, you can go look at
your graphs and get real-time feedback on what it's doing. If you
don't have that, all you get is a frustrating silence.
Now, the object updater tells you all of its stats (successes,
failures, quarantines due to bad pickles, unlinks, and errors), and it
tells you incremental progress every five minutes. The logging at the
end of a pass remains and has been expanded to also include all stats.
Also included is a small change to what counts as an error: unmounted
drives no longer do. The goal is that only abnormal things count as
errors, like permission problems, malformed filenames, and so
on. These are things that should never happen, but if they do, may
require operator intervention. Drives fail, so logging an error upon
encountering an unmounted drive is not useful.
Change-Id: Idbddd507f0b633d14dffb7a9834fce93a10359ab

Change-Id: I98e7e55839e5697d975af0a4756fd09933af1c5f

Change-Id: Ia21f8743bfd745f2579db8658624f888461c2cc2

...and update the object-expirer man page.
Change-Id: Idca1b8e3b7d5b40481af0d60477510e2557b88c0

There was incorrect path starting /home/swift, but /home/<your-user-name> is correct for common users.
Change-Id: Ia81b2119c87dd88417428e55c82dac1ab7c028b3
Closes-Bug: 1741378

Change-Id: Ib2483444d3999e13ba83ca2edd3a8ef8e5c48548

With a sufficiently undispersed ring it's possible to move an entire
replicas worth of parts and yet the value of dispersion may not get any
better (even though in reality dispersion has dramatically improved).
The problem is dispersion will currently only represent up to one whole
replica worth of parts being undispersed.
However with EC rings it's possible for more than one whole replicas
worth of partitions to be undispersed, in these cases the builder will
require multiple rebalance operations to fully disperse replicas - but
the dispersion value should improve with every rebalance.
N.B. with this change it's possible for rings with a bad dispersion
value to measure as having a significantly smaller dispersion value
after a rebalance (even though they may not have had their dispersion
change) because the total amount of bad dispersion we can measure has
been increased but we're normalizing within a similar range.
Closes-Bug: #1697543
Change-Id: Ifefff0260deac0c3e8b369a1e158686c89936686

In current manpage docs, some of daemon names for concurrency
explanation is wrong.
This patch fixes the daemon names.
Change-Id: I2a505c9590ee3a3a7e37e8d949a10db36206faec