openstack/swift - swift - OpenDev: Free Software Needs Free Tools

There was incorrect path starting /home/swift, but /home/<your-user-name> is correct for common users.
Change-Id: Ia81b2119c87dd88417428e55c82dac1ab7c028b3
Closes-Bug: 1741378

Change-Id: Ib2483444d3999e13ba83ca2edd3a8ef8e5c48548

With a sufficiently undispersed ring it's possible to move an entire
replicas worth of parts and yet the value of dispersion may not get any
better (even though in reality dispersion has dramatically improved).
The problem is dispersion will currently only represent up to one whole
replica worth of parts being undispersed.
However with EC rings it's possible for more than one whole replicas
worth of partitions to be undispersed, in these cases the builder will
require multiple rebalance operations to fully disperse replicas - but
the dispersion value should improve with every rebalance.
N.B. with this change it's possible for rings with a bad dispersion
value to measure as having a significantly smaller dispersion value
after a rebalance (even though they may not have had their dispersion
change) because the total amount of bad dispersion we can measure has
been increased but we're normalizing within a similar range.
Closes-Bug: #1697543
Change-Id: Ifefff0260deac0c3e8b369a1e158686c89936686

In current manpage docs, some of daemon names for concurrency
explanation is wrong.
This patch fixes the daemon names.
Change-Id: I2a505c9590ee3a3a7e37e8d949a10db36206faec

Cleanup for docs and docstrings.
Related-Change: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Change-Id: Ie8de0565dfaca5bd8a5693a75e6ee14ded5b7161

Add a symbolic link ("symlink") object support to Swift. This
object will reference another object. GET and HEAD
requests for a symlink object will operate on the referenced object.
DELETE and PUT requests for a symlink object will operate on the
symlink object, not the referenced object, and will delete or
overwrite it, respectively.
POST requests are *not* forwarded to the referenced object and should
be sent directly. POST requests sent to a symlink object will
result in a 307 Error.
Historical information on symlink design can be found here:
https://github.com/openstack/swift-specs/blob/master/specs/in_progress/symlinks.rst.
https://etherpad.openstack.org/p/swift_symlinks
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Signed-off-by: Thiago da Silva <thiago@redhat.com>

Fix what appears to be a cut and paste error.
Change-Id: Iccf97ebbf75c8f97095a4493ea6a8beb074df099

Related-Change: I3c2e569f0c44168333251bb58bab4b5582e15a45
Change-Id: I9776c0919164e48ac445eacf7d897a23ef8e4572

TrivialFix
Change-Id: Iacd341a1f3dbf43c6fc2ebcf546aed54f866face

* Fix formatting error in SAIO docs so that list is displaying
 correct
Change-Id: If88869290252a2c86f60fdf89d78ef4a418ae23d

Currently, our integrity checking for objects is pretty weak when it
comes to object metadata. If the extended attributes on a .data or
.meta file get corrupted in such a way that we can still unpickle it,
we don't have anything that detects that.
This could be especially bad with encrypted etags; if the encrypted
etag (X-Object-Sysmeta-Crypto-Etag or whatever it is) gets some bits
flipped, then we'll cheerfully decrypt the cipherjunk into plainjunk,
then send it to the client. Net effect is that the client sees a GET
response with an ETag that doesn't match the MD5 of the object *and*
Swift has no way of detecting and quarantining this object.
Note that, with an unencrypted object, if the ETag metadatum gets
mangled, then the object will be quarantined by the object server or
auditor, whichever notices first.
As part of this commit, I also ripped out some mocking of
getxattr/setxattr in tests. It appears to be there to allow unit tests
to run on systems where /tmp doesn't support xattrs. However, since
the mock is keyed off of inode number and inode numbers get re-used,
there's lots of leakage between different test runs. On a real FS,
unlinking a file and then creating a new one of the same name will
also reset the xattrs; this isn't the case with the mock.
The mock was pretty old; Ubuntu 12.04 and up all support xattrs in
/tmp, and recent Red Hat / CentOS releases do too. The xattr mock was
added in 2011; maybe it was to support Ubuntu Lucid Lynx?
Bonus: now you can pause a test with the debugger, inspect its files
in /tmp, and actually see the xattrs along with the data.
Since this patch now uses a real filesystem for testing filesystem
operations, tests are skipped if the underlying filesystem does not
support setting xattrs (eg tmpfs or more than 4k of xattrs on ext4).
References to "/tmp" have been replaced with calls to
tempfile.gettempdir(). This will allow setting the TMPDIR envvar in
test setup and getting an XFS filesystem instead of ext4 or tmpfs.
THIS PATCH SIGNIFICANTLY CHANGES TESTING ENVIRONMENTS
With this patch, every test environment will require TMPDIR to be
using a filesystem that supports at least 4k of extended attributes.
Neither ext4 nor tempfs support this. XFS is recommended.
So why all the SkipTests? Why not simply raise an error? We still need
the tests to run on the base image for OpenStack's CI system. Since
we were previously mocking out xattr, there wasn't a problem, but we
also weren't actually testing anything. This patch adds functionality
to validate xattr data, so we need to drop the mock.
`test.unit.skip_if_no_xattrs()` is also imported into `test.functional`
so that functional tests can import it from the functional test
namespace.
The related OpenStack CI infrastructure changes are made in
https://review.openstack.org/#/c/394600/.
Co-Authored-By: John Dickinson <me@not.mn>
Change-Id: I98a37c0d451f4960b7a12f648e4405c6c6716808

An SLO PUT requires that we HEAD every referenced object; as a result, it
can be a very time-intensive operation. This makes it difficult as a
client to differentiate between a proxy-server that's still doing work and
one that's crashed but left the socket open.
Now, clients can opt-in to receiving heartbeats during long-running PUTs
by including the query parameter
 heartbeat=on
With heartbeating turned on, the proxy will start its response immediately
with 202 Accepted then send a single whitespace character periodically
until the request completes. At that point, a final summary chunk will be
sent which includes a "Response Status" key indicating success or failure
and (if successful) an "Etag" key indicating the Etag of the resulting SLO.
This mechanism is very similar to the way bulk extractions and deletions
work, and even the way SLO behaves for ?multipart-manifest=delete requests.
Note that this is opt-in: this prevents us from sending the 202 response
to existing clients that may mis-interpret it as an immediate indication
of success.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Related-Bug: 1718811
Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3

Change-Id: I7f070956d8b996db798837392adfca4483067aea

The overview_policies doc makes reference to an `alias` option when in
fact the option is `aliases`.
The sample storage policy snippet is correct, it's just incorrect when
listing the possible options.
This change changes the listed option to `aliases`.
Change-Id: Iddf0f19f4d50819ff6abd46e6a1156dc8e4a451d

Change-Id: Ic7fc6ddca2ab564b31156fa84b362bc9963825f1
Closes-Bug: #1607025 

This commit replaces boolean replication_one_per_device by an integer
replication_concurrency_per_device. The new configuration parameter is
passed to utils.lock_path() which now accept as an argument a limit for
the number of locks that can be acquired for a specific path.
Instead of trying to lock path/.lock, utils.lock_path() now tries to lock
files path/.lock-X, where X is in the range (0, N), N being the limit for
the number of locks allowed for the path. The default value of limit is
set to 1.
Change-Id: I3c3193344c7a57a8a4fc7932d1b10e702efd3572

Commit 250da37a removed bin/swift-temp-url, but left the man page.
Change-Id: Ic518f23678e3c3134f02a46a51a2bcb90d92bdc2

Related-Change: I0760ce149e6d74f2b3f1badebac3e36da1ab7e77
Change-Id: I129de42f91d7924c7bcb9952f17fe8a1a10ae219

Updated the comments of reviewers.
Change-Id: I0760ce149e6d74f2b3f1badebac3e36da1ab7e77
Closes-Bug: #1607026 

Change-Id: I98643d6acf248840a8360f31e446bc8ecb834898

Change-Id: I2c3eea783321338316eecf467d30ba0b3217256c
Related-Bug: #1514528 

Change-Id: Iab1beff4899d096936c0e5915f3ec32364b3e517
Closes-Bug: #1559347 

Closes-Bug: #1719095
Change-Id: I8051895987bf72c8095e72b5a521042a13993174
Signed-off-by: Thiago da Silva <thiago@redhat.com>

It was deprecated and we discussed on this topic in Denver PTG
for Queen cycle. Main motivation for this work is that deprecated
post_as_copy option and its gate blocks future symlink work.
Change-Id: I411893db1565864ed5beb6ae75c38b982a574476

Make some json -> (text, xml) stuff in a common module, reference that in
account/container servers so we don't break existing clients (including
out-of-date proxies), but have the proxy controllers always force a json
listing.
This simplifies operations on listings (such as the ones already happening in
decrypter, or the ones planned for symlink and sharding) by only needing to
consider a single response type.
There is a downside of larger backend requests for text/plain listings, but
it seems like a net win?
Change-Id: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d

Update the doc link brought by the doc migration.
Although we had some effort to fix these, it still left lots of bad
doc link, I separate these changes into 3 patches aim to fix all of
these, this is the 2st patch for doc/manpages.
Change-Id: Id426c5dd45a812ef801042834c93701bb6e63a05

Update the doc link brought by the doc migration.
Although we had some effort to fix these, it still left lots of bad
doc link, I separate these changes into 3 patches aim to fix all of
these, this is the 3rd patch for doc/source/install.
Change-Id: I1b0c12cd5f893f1a84d12782ddc39f6d06beb2aa

- saio describes both 14.04 and 16.04 procedure
- currently we're testing on 16.04 (xenial) envrionment on the gate
Remaining task (probably another work):
- review the installation guide which adjusts to the ubuntu 14.04 LTS
Change-Id: Id690a1deabeb24bfc1af3ba3a3019794fe4b8eb9

If a number of DLO segments is larger than container listing limit,
Content-Length header will not be included in GET or HEAD response.
However, this fact is not explained in document of large objects.
This patch add explanation about this fact to the document.
Change-Id: Ia45fad05797f38fa8b6b0ed917b4f9d7fb337149
Closes-Bug: 1680219

Update the doc link brought by the swift doc migration.
Although we had some effort to fix these before, it still left lots
of bad doc link, I separate these changes into 3 patches aim to fix
all of these, this is the 1st patch for doc/manpages.
Change-Id: Ib49696706e61bbd36ae56b15b1d94aa4ce84531c

At patch [1], container sync daemon is updated to use internal client.
However, the "Configuring Container Sync" section [2] of swift docs doesn't
updated for this change.
This patch adds an explanation about the change.
[1]: https://review.openstack.org/#/c/452773/
[2]: http://docs.openstack.org/developer/swift/overview_container_sync.html#configuring-container-sync
Change-Id: Ia1aabeab250cbe27414d7877138ad38708f5ce3f
Closes-Bug: 1446721

This isn't actually used (and in swift is commented out already)
and is a leftover from a thing we did about seven years ago.
Change-Id: I9889bcfd29054f14679ae7430b077ad3afb25b98