openstack/swift - swift - OpenDev: Free Software Needs Free Tools

This patch standardizes the CONTRIBUTING.rst file and adds the
required doc/source/contributor/contributing.rst
Swift already had a detailed CONTIRBUTING.rst and an informative
REVIEW_GUIDELINES.rst in the root of the repo. So we are also pulling
them into the contributor documentation so they can not only be easily
found in the checked repo but in the online documentation.
Change-Id: I4c84efbe50eb25ab922c9d6b69198dae341af48b

And stop sending WSGI strings on py3.
Change-Id: I9b769e496aa7c8ed5862c2d7310f643838328084
Closes-Bug: #1853654 

Real AWS only includes an empty delimiter element when doing a
version-aware listing.
Change-Id: Id246a157c576eac93375be084ada3740f1e09793
Closes-Bug: #1853663 

* Stop specifying logbufs=8; that's the default
* Stop including nodiratime with noatime; the latter implies the former
Nothing wrong with being explicit, I suppose, but may as well keep the
mount options to what we can easily explain: we want noatime because
Swift does not use atime, so we don't want to lose any performance to
tracking atime.
Change-Id: I1e52b4368ad7eb375964eee5132bc50297536355

Update doc examples to use explicit UUID of devices, instead of using
unstable device names.
Change-Id: I3a2eb7bbe4b4091d2567897904d939df1df6b251
Closes-Bug: #1817966 

Change-Id: I48ebbb38427612a52c36078586af7b9780b5ea37

Translate AWS S3 Object Versioning API requests to native Swift Object
Versioning API, speficially:
 * bucket versioning status
 * bucket versioned objects listing params
 * object GETorHEAD & DELETE versionId
 * multi_delete versionId
Change-Id: I8296681b61996e073b3ba12ad46f99042dc15c37
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

Specify that the relink command must be run as the same user than the
daemon processes to avoid creating files that are not manipulable by the
server/replicator/...
Change-Id: If23592855db5f5bb0ec1e7c679de15769fd86871

Change-Id: I1611e34846e586703e9d3709fa64e8df41f2d685

Users have complained for a while that Swift's ETags don't match the
expected RFC formats. We've resisted fixing this for just as long,
worrying that the fix would break innumerable clients that expect the
value to be a hex-encoded MD5 digest and *nothing else*.
But, users keep asking for it, and some consumers (including some CDNs)
break if we *don't* have quoted etags -- so, let's make it an option.
With this middleware, Swift users can set metadata per-account or even
per-container to explicitly request RFC compliant etags or not. Swift
operators also get an option to change the default behavior
cluster-wide; it defaults to the old, non-compliant format.
See also:
 - https://tools.ietf.org/html/rfc2616#section-3.11
 - https://tools.ietf.org/html/rfc7232#section-2.3
Closes-Bug: 1099087
Closes-Bug: 1424614
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I380c6e34949d857158e11eb428b3eda9975d855d

This patch adds a new object versioning mode. This new mode provides
a new set of APIs for users to interact with older versions of an
object. It also changes the naming scheme of older versions and adds
a version-id to each object.
This new mode is not backwards compatible or interchangeable with the
other two modes (i.e., stack and history), especially due to the changes
in the namimg scheme of older versions. This new mode will also serve
as a foundation for adding S3 versioning compatibility in the s3api
middleware.
Note that this does not (yet) support using a versioned container as
a source in container-sync. Container sync should be enhanced to sync
previous versions of objects.
Change-Id: Ic7d39ba425ca324eeb4543a2ce8d03428e2225a1
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>

This allows static symlinks to be synced before their target. Dynamic
symlinks could already be synced even if target object has not been
synced, but static links previously required that target object existed
before it can be PUT. Now, have container_sync middleware plumb in an
override like it does for SLO.
Change-Id: I3bfc62b77b247003adcee6bd4d374168bfd6707d

Change-Id: I98ebe15353d675ca00fee387003bf6572ac385e6

This is to fix the duplicated words issue, like:
 * "both of which which must be stripped off"
 * "In addition the the values set"
 and so on
Change-Id: Id3d84281f15815b4185c76874575e91a3589981b

If we move it to constraints it's more globally accessible in our code,
but more importantly it's more obvious to ops that everything breaks if
you try to mis-configure different values per-service.
Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee

Change-Id: I89f3a4b5a3a8c160afb298aad726acce09c65265

Change-Id: Ib54a65a920fc335c1e0a496cb91c0de3003e2cc5

We get a handful more passing tests that way, following their move to boto3.
Change-Id: I73e9c38bde00a7117cec97e98f013e86350aa5be

Change-Id: I775a03e0ba1e10982eb6f7ef52be773c8831b1ec

Change-Id: I3833843cd8d23c498a2afe6c68a3f0afe26343c0

Swift servers can now be seamlessly reloaded by sending them a SIGUSR1
(instead of a SIGHUP). The server forks off a synchronized child to
wait to close the old listen socket(s) until the new server has started
up and bound its listen socket(s). The new server is exec'ed from the
old one so its PID doesn't change. This makes Systemd happier, so a
ReloadExec= stanza can now be used.
The seamless part means that incoming connections will alwyas get
accepted either by the old server or the new one. This eliminates
client-perceived "downtime" during server reloads, while allowing the
server to fully reload, re-reading configuration, becoming a fresh
Python interpreter instance, etc. The SO_REUSEPORT socket option has
already been getting used, so nothing had to change there.
This patch also includes a non-invasive fix for a current eventlet bug;
see https://github.com/eventlet/eventlet/pull/590
That bug prevents a SIGHUP "reload" from properly servicing existing
requests before old worker processes close sockets and exit. The
existing probtests missed this, but the new ones, in this patch, caught
it.
New probe tests cover both old SIGHUP "reload" behavior as well as the
new SIGUSR1 seamless reload behavior.
Change-Id: I3e5229d2fb04be67e53533ff65b0870038accbb7

During a PUT of an object, the proxy instanciates one Putter per
object-server that will store data (either the full object or a
fragment, depending on the storage policy). Each Putter is owning a
Queue that will be used to bufferize data chunks before they are
written to the socket connected to the object-server. The chunks are
moved from the queue to the socket by a greenthread. There is one
greenthread per Putter. If the client is uploading faster than the
object-servers can manage, the Queue could grow and consume a lot of
memory. To avoid that, the queue is bounded (default: 10). Having a
bounded queue also allows to ensure that all object-servers will get
the data at the same rate because if one queue is full, the
greenthread reading from the client socket will block when trying to
write to the queue. So the global rate is the one of the slowest
object-server.
The thing is, every operating system manages socket buffers for incoming
and outgoing data. Concerning the send buffer, the behavior is such that
if the buffer is full, a call to write() will block, otherwise the call
will return immediately. It behaves a lot like the Putter's Queue,
except that the size of the buffer is dynamic so it adapts itself to the
speed of the receiver.
Thus, managing a queue in addition to the socket send buffer is a
duplicate queueing/buffering that provides no interest but is, as shown
by profiling and benchmarks, very CPU costly.
This patch removes the queuing mecanism. Instead, the greenthread
reading data from the client will directly write to the socket. If an
object-server is getting slow, the buffer will fulfill, blocking the
reader greenthread. Benchmark shows a CPU consumption reduction of more
than 30% will the observed rate for an upload is increasing by about
45%.
Change-Id: Icf8f800cb25096f93d3faa1e6ec091eb29500758

The Swift AIO documentation is out of date for CentOS and Fedora.
This patch updates the documentation to ensure that the instructions are
clear and accurate.
It also cleans up a few other sections along the way and adds some new
headings to make it easier to read. Some explanatory notes are added,
such as the need for XFS storage and test directories. XFS tmp loopback
device is moved out of the user's home directory to /srv to match
storage.
Change-Id: Ieb0341536b7149c99139a1cf620828eba25f4bc6
Signed-off-by: Chris Smart <chris.smart@humanservices.gov.au>

Change-Id: I948d65d0d3b58fcb13f8141fe095ccad4b8f1425
Closes-Bug: #1848485 

This patch adds a `pdf-docs` tox target that will build
PDF versions of our docs. As per the Train community goal:
 https://governance.openstack.org/tc/goals/selected/train/pdf-doc-generation.html
Add sphinxcontrib-svg2pdfconverter to doc/requirements.txt
to convert our SVGs.
Story: 2006122
Task: 35515
Change-Id: I26cefda80d3234df68d7152b404e0a71da74ab90

Change-Id: I94050c40585b397a9f7bab1e48650b89f70ab24d

Set copyright year and html_last_updated_fmt to SOURCE_DATE_EPOCH if
it's set. See https://reproducible-builds.org/specs/source-date-epoch/
This patch make build reproducible, see https://reproducible-builds.org/
Change-Id: I730a8265ca2c70c639ef77a613908e84eb738b70

Small changes, but helpful, mostly.
Backport: stein rocky
Change-Id: Ic4b6524d7804d2f74b2973b6acdb9e2679209cd4

Change-Id: I3fee824e4fdc21aac21a6bb97f24a82d8ea3577d

The URL is object-store, not object-storage. Fix one occurence.
Change-Id: I08ed3b19cbe0024fd67a6fb7a7a6db0d9084976b

The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that the Swift API lives at /object-store and not /object-storage.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: Ie38357e4c278335c35d186708573bb6bdabaa012

Beginning with the Queens release, the keystone install guide
recommends running all interfaces on the same port.This patch
updates the swift install guide to reflect that change
Change-Id: Id00cfd2c921da352abdbbbb6668b921f3cb31a1a
Closes-bug: #1754104 

XFS no longer supports nobarrier mount option.
It has been deprecated for a long time[1] and removed in
recent kernel versions resulting in an error when trying to
mount: "kernel: XFS (loop0): unknown mount option [nobarrier]."
[1] - https://patchwork.kernel.org/patch/9486549/
Change-Id: Iaa9208fb20545ae9ac990f0e180899108d983123

Change-Id: Ie759c4092408a210d32f6457142457087343d33a

...to pick up many improvements, including the return of table borders.
Change-Id: I166211b690b08521171b489582fa419d756b1972

This precludes us from landing anything in the gate, because we
treat Sphinx warnings as errors in OpenStack.
Specifically, 3 warnings are fixed:
/home/zuul/src/opendev.org/openstack/swift/swift/common/middleware\
/s3api/etree.py:docstring of\
 swift.common.middleware.s3api.etree.Element:1:\
Inline strong start-string without end-string.
duplicate object description of\
 swift.common.middleware.versioned_writes, other instance in\
 middleware, use :noindex: for one of them
duplicate object description of\
 swift.common.ring.composite_builder, other instance in\
 overview_ring, use :noindex: for one of them
Change-Id: I5e0bd61a301d46b2674b4605882462575b635220