openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Reseller admins can set new headers on accounts like
 X-Account-Quota-Bytes-Policy-<policy-name>: <quota>
This may be done to limit consumption of a faster, all-flash policy, for
example.
This is independent of the existing X-Account-Meta-Quota-Bytes header, which
continues to limit the total storage for an account across all policies.
Change-Id: Ib25c2f667e5b81301f8c67375644981a13487cfe

* Get rid of a bunch of accidental blockquote formatting
* Always declare a lexer to use for ``.. code::`` blocks
Change-Id: I8940e75b094843e542e815dde6b6be4740751813

We've known this would eventually be necessary for a while [1], and
way back in 2017 we started seeing SHA-1 collisions [2].
[1] https://www.schneier.com/blog/archives/2012/10/when_will_we_se.html
[2] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
UpgradeImpact:
==============
"sha1" has been removed from the default set of `allowed_digests` in the
tempurl middleware config. If your cluster still has clients requiring
the use of SHA-1,
- explicitly configure `allowed_digests` to include "sha1" and
- encourage your clients to move to more-secure algorithms.
Depends-On: https://review.opendev.org/c/openstack/tempest/+/832771
Change-Id: I6e6fa76671c860191a2ce921cb6caddc859b1066
Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Closes-Bug: #1733634 

Change-Id: I4ee5a52ec9fb5f1920cd6869f6b1245c3787391c

The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that the Swift API lives at /object-store and not /object-storage.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: Ie38357e4c278335c35d186708573bb6bdabaa012

Change-Id: Ic7025ba79eadb39f75dd03135fae3326138b3ded

These files are imported (and very lightly edited) from the old
ocata user-guide. It has a few other swift-related docs that seemed
more duplacative of what we already have, but these seem to fill
existing gaps in our docs.
Change-Id: Ib00bf6992327f15f271120dc5dbc86a4a235baec

Curly quotes(Chinese punctuation) usually input from Chinese input method.
When read from english context, it makes some confusion.
Change-Id: Ibd50299ee287c56ec4759ea8ff53d47d006144f8

Change-Id: I9a04cb2fbe61e1fbd8185ab2fac9abbcea4d55cc

Add a symbolic link ("symlink") object support to Swift. This
object will reference another object. GET and HEAD
requests for a symlink object will operate on the referenced object.
DELETE and PUT requests for a symlink object will operate on the
symlink object, not the referenced object, and will delete or
overwrite it, respectively.
POST requests are *not* forwarded to the referenced object and should
be sent directly. POST requests sent to a symlink object will
result in a 307 Error.
Historical information on symlink design can be found here:
https://github.com/openstack/swift-specs/blob/master/specs/in_progress/symlinks.rst.
https://etherpad.openstack.org/p/swift_symlinks
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Signed-off-by: Thiago da Silva <thiago@redhat.com>

An SLO PUT requires that we HEAD every referenced object; as a result, it
can be a very time-intensive operation. This makes it difficult as a
client to differentiate between a proxy-server that's still doing work and
one that's crashed but left the socket open.
Now, clients can opt-in to receiving heartbeats during long-running PUTs
by including the query parameter
 heartbeat=on
With heartbeating turned on, the proxy will start its response immediately
with 202 Accepted then send a single whitespace character periodically
until the request completes. At that point, a final summary chunk will be
sent which includes a "Response Status" key indicating success or failure
and (if successful) an "Etag" key indicating the Etag of the resulting SLO.
This mechanism is very similar to the way bulk extractions and deletions
work, and even the way SLO behaves for ?multipart-manifest=delete requests.
Note that this is opt-in: this prevents us from sending the 202 response
to existing clients that may mis-interpret it as an immediate indication
of success.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Related-Bug: 1718811
Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3

Update the doc link brought by the doc migration.
Although we had some effort to fix these, it still left lots of bad
doc link, I separate these changes into 3 patches aim to fix all of
these, this is the 2st patch for doc/manpages.
Change-Id: Id426c5dd45a812ef801042834c93701bb6e63a05

If a number of DLO segments is larger than container listing limit,
Content-Length header will not be included in GET or HEAD response.
However, this fact is not explained in document of large objects.
This patch add explanation about this fact to the document.
Change-Id: Ia45fad05797f38fa8b6b0ed917b4f9d7fb337149
Closes-Bug: 1680219

Change-Id: I2f63ce16ae67277504a47e68c7dca3c3412fe92c

Change-Id: Iac24369910464cb766fe7d5e6c15120d147930a7

Change-Id: I6e200558b75ac539b59b492d13c36702443efc89

Change-Id: Ib7d2c16a755ae1faca5a371d5dae1e143110178f

With this commit, the tempurl middleware accepts (besides
the traditional unix timestamps) also timestamps according
to the format '%Y-%m-%dT%H:%M:%SZ' (one acceptable form of ISO 8601).
The idea is to make the tempurls more user-friendly,
and has been formulated here:
Change-Id: I346a0241060a9559d178b30e60c957792bbeb9f0
Implements: blueprint human-readable-tempurl-timestamp

Let's remove the reference to swift-temp-url since
it has been deprecated and add a link to the swift client.
Change-Id: I70d64bf90f23a0f48b238ae6a99ab86f87d028a1
Signed-off-by: Thiago da Silva <thiago@redhat.com>

Change-Id: I7442f2056d7dbbd83d514eb21564697af99e2068

With a multipart-manifest PUT request, if client sends the md5 of the
segments' etags, a 422 Unprocessable Entity response is returned. This
patch fixes that and confirms the etag
Change-Id: I4598a2a3f16ca8727bb07bbb6d8efcfcae777796
Closes-Bug: #1213200
Co-Authored-By: Tim Burke <tim@swiftstack.com>

Previously, we still required that clients send "etag" and "size_bytes"
keys in their segment definitions. This was done as a way to guard
against typos leading to an accidental lack of verification.
However, typos should already be caught when we check for extra keys. As
a result, the only truly required key is "path".
Change-Id: Ie1d8691115f8c68b5a3f3b59317cdab59f9a3fca

The middleware now allows the usage of signatures with a prefix-based
scope. A prefix-based signature grants access to all objects which share
the same prefix. This avoids the creation of a large amount of signatures,
when a whole container or pseudofolder is shared.
Please see spec: https://review.openstack.org/#/c/199607/
Change-Id: I03b68eb74dae6196b5e63e711ef642ff7d2cfdc9

Many other OpenStack services use a `[X-]OpenStack-Request-Id` header
to return a unique identifier for the request. Swift will now return
`X-Trans-Id` as well as `X-Openstack-Request-Id`.
Change-Id: I56cd4738808b99c0a08463f83c100be51a62db05
Closes-Bug: #1572786 

Change-Id: I44743fbb9bcbce3a50ed6770264ba0f4b17803d7

Now, instead of saying
 X-Versions-Location: <container>
 X-Versions-Mode: history
clients should just say
 X-History-Location: <container>
Since we've never had a release featuring a user-settable
X-Versions-Mode header, support may be dropped and that is now ignored.
Change-Id: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9

This patch update the outdated link reference inside the doc.
Change-Id: I9c5aabe869226489f361ff637e9dec731bbc3d7d

Add a note to the docstring that it is required to add a config section
to the proxy-server.conf and an entry to the pipeline to support history
mode.
Closes-Bug: 1619261
Change-Id: I888485ab4ece6f47db081a4d58c1aab24ce72a8a

Currently when using fast-post, the manifest is updated with the given
'x-object-manifest' header on a POST. If no such header is supplied,
then the manifest will change to a regular object.
This is not currently true when using post-as-copy.
This patch changes the DLO POST using post-as-copy behavior to match
that of using fast-post. It was also documented that
'x-object-manifest' must be provided on a POST to a manifest file.
Change-Id: Ie1143ab1a2c8f8c21e258a36badbff5d947769d4
Closes-bug: 1612991

This change introduces the concept of a "versioning mode" for
versioned_writes. The following modes are supported:
 * stack
 When deleting, check whether any previous versions exist in the
 versions container. If none is found, the object is deleted. If the
 most-recent version in the versions container is not a delete
 marker, it is copied into the versioned container (overwriting the
 current version if one exists) and then deleted from the versions
 container. This preserves the previous behavior.
 If the most-recent version in the versions container is a delete
 marker and a current version exists in the versioned container, the
 current version is deleted. If the most-recent version in the
 versions container is a delete marker and no current version exists
 in the versioned container, we copy the next-most-recent version
 from the versions container into the versioned container (assuming
 it exists and is not a delete marker) and delete both the
 most-recent version (i.e., the delete marker) and the just-copied
 next-most-recent version from the versions container.
 With this mode, DELETEs to versioned containers "undo" operations
 on containers. Previously this was limited to undoing PUTs, but now
 it will also undo DELETEs performed while in "history" mode.
 * history
 When deleting, check whether a current version exists in the
 versioned container. If one is found, it is copied to the versions
 container. Then an empty "delete marker" object is also put into the
 versions container; this records when the object was deleted.
 Finally, the original current version is deleted from the versioned
 container. As a result, subsequent GETs or HEADs will return a 404,
 and container listings for the versioned container do not include
 the object.
 With this mode, DELETEs to versioned containers behave like DELETEs
 to other containers, but with a history of what has happened.
Clients may specify (via a new X-Versions-Mode header) which mode a
container should use. By default, the existing "stack" mode is used.
Upgrade consideration:
======================
Clients should not use the "history" mode until all proxies in the
cluster have been upgraded. Attempting to use the "history" mode during
a rolling upgrade may result in some requests being served by proxies
running old code (which necessarily uses the "stack" mode), leading to
data loss.
Change-Id: I555dc17fefd0aa9ade681aa156da24e018ebe74b

If the curl command is used exactly as in the help, the ampersand
in the signature is interpreted as an operator and the curl
command breaks. I am aware of developers who have wasted a lot of
time because of this.
Change-Id: I6468c9a098b56db8242a2cf2c23b7a4857bd8574

Change-Id: I7f6f4ffb7033d2a321362c91e05f1576847939b0

Change-Id: I187fb8fc8d10f950bbebc586999eced0bc117432

Added a link to the API reference (where headers and query
strings are documented; click the "detail" button to see).
Also added a reference to Swift section of the OpenStack end
user guide. This contains some additional details about the API.
No attempt was made to reconcile duplicate information. Instead
this patch links documents that might otherwise be overlooked.
However, I fixed text, originally in a table, that had become
garbled in a prior patch.
Change-Id: I0910cbeb0c8bffc00e510f35585603e7b7a67790

Also fix typo in the api doc.
Follow-on to change I5eb655360ac95042877da26d18707aebc11c02f6
Change-Id: Ic1ea63c74a0a3b90a505865fba8fcfac584d0825

This change adds the ability to tell the container or account server to
reverse their listings. This is done by sending a reverse=TRUE_VALUE,
Where TRUE_VALUE is one of the values true can be in common/utils:
 TRUE_VALUES = set(('true', '1', 'yes', 'on', 't', 'y'))
For example:
 curl -i -X GET -H "X-Auth-Token: $TOKEN" $STORAGE_URL/c/?reverse=on
I borrowed the swapping of the markers code from Kevin's old change,
thanks Kevin. And Tim Burke added some real nuggets of awesomeness.
DocImpact
Co-Authored-By: Kevin McDonald <kmcdonald@softlayer.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Implements: blueprint reverse-object-listing
Change-Id: I5eb655360ac95042877da26d18707aebc11c02f6

Change-Id: I3c708bacd468e923e574bdd5afc7acc58b072159

The doc for these sections was missing because of an rst error - the
source is there in rst file but didn't make it into the html output.
Add doc for per_diff and max_diffs in account and container doc sections.
Also, fix a bunch of other sphinx build errors and most of the warnings.
Change-Id: If9ed2619b2f92c6c65a94f41d8819db8726d3893

Update links to the Cloud Admin Guide after the
RST conversion of that book altered URLs.
Change-Id: I899f8938498b744e62887968a65e58c00ef27f1b

The content about Object Storage HTTP requests constraints seems to
be a table but broken, this patch fixes it into correct rst table format.
Change-Id: I1b4c62da3e6d91add3ee0218707c1628c8f04b33

Inline markup cannot be nested in RST.
Fixing markups by using a single option for the whole line and making
some minor error corrections.
Change-Id: I42bfd7dfe5c93a69436ecc5d154f2e61ca83fa82

Fixed Formatting error in ``action="SWIFT_URL``" => action="``SWIFT_URL``"
Change-Id: Id461129ef7f5b6412f94d36920c942a4181c0eb7
Closes-Bug: #1464740 

This change is in support of Composite Tokens and Service Accounts
(see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/
service_token.html)
During coding, minor changes were made compared to the original
specification. See https://review.openstack.org/138771 for these changes.
DocImpact
Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30