openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Cleans up some auditor tests added in the Related-Change.
Make auditor NOT call invalidate_hash for a reclaimable
tombstone in a zero bytes file process, so that
invalidate_hash is only called once per reclaimable
tombstone per auditor cycle.
Previously each execution of 'swift-init object-auditor once'
would result in two identical entries being appended to
hashes.invalid for each reclaimable tombstone. With this change
that unnecessary duplication is removed.
Related-Change: I3e99dc702d55a7424c6482969e03cb4afac854a4
Change-Id: I5dfaa8d74c07ca8a494c29159c1a2bed39499613

I don't think this is a real bug - just that the mocked iter wasn't
closing it subiters like the real iter does.
Change-Id: I44c8159f9eea8737bc86b6c7eb59a512e57e86c1

Now, instead of saying
 X-Versions-Location: <container>
 X-Versions-Mode: history
clients should just say
 X-History-Location: <container>
Since we've never had a release featuring a user-settable
X-Versions-Mode header, support may be dropped and that is now ignored.
Change-Id: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9

- Call invalidate_hash in auditor for reclaimable tombstones
- assert changed auditor behavior with a unit test
- driveby test: assert get_hashes behavior with a unit test
Co-Authored-By: Pete Zaitcev <zaitcev@redhat.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Closes-Bug: #1301728
Change-Id: I3e99dc702d55a7424c6482969e03cb4afac854a4

test_GET_HEAD_with_fragment_preferences seemed to fail on ocassion on
my machine; the test seemed to be assuming the order of dictionary
keys in json output sent in headers; the fix was convert back to a
dictionary when making the assertion on the context of the json.
Change-Id: I1ea1b734c2a9fb12b8f59262bb1229421803e48e
Closes-Bug: #1625956 

Clarify Content-Type header definition for listings.
Distinguish between request and response definitions for
X-Account-Meta-Temp-URL-Key* headers.
Insert X-Container-Meta-Quota-* headers missing from some
request/response definitions.
Improve X-Container-Meta-Access-Control-Expose-Headers
parameter formatting.
Distinguish between request and response definitions for
X-Container-Meta-Temp-URL-Key* headers.
Co-Authored-By: Christian Schwede <cschwede@redhat.com>
Change-Id: I8fc7610395973b520aa9ddd72c94e1eb75f602cd
Related-Change: I315b4e550b7d10880fbc00fce9311127ba609c2d

If we're going to talk about replacing an object, we should use the
same object name as the previous example.
Including a non-zero content-length on PUT but not providing a body
will timeout.
Not including the '-' in '-H' will make curl complain:
 Rebuilt URL to: H/
 * Could not resolve host: H
 * Closing connection 1
201 Created responses have content-length of zero, not 116.
Change-Id: Ifd878559ee4036e4893221c7968f53021f38e236

Change-Id: Iab839a917e407ce8119f0c6162751f11136cbf5b

Move the account listing sample responses to follow the sample
requests, and to precede the request/response parameter definitions to
be consistent with other parts of the doc.
Related-Change: I315b4e550b7d10880fbc00fce9311127ba609c2d
Change-Id: Ia20acacd238db4a423b8cd89af1658222b4c5828

Move repeated test re metadata header syntax to an include
file and make it be rendered as a note.
Also make already included text about metadata header value
encoding be a note.
Change-Id: I4795836587492954ad24dd5baaa5d668746d6040

This is *only* sysmeta, to ensure we neither change the semantics of the
response (which could happen if we included allowed_headers) nor reveal
previously-written metadata to write-only clients (which could happen if
we ever get updateable object metadata and sent back user meta).
We could conceivably send back transient sysmeta, though it seems
better to wait for a use-case that demands it.
Change-Id: I04fe0b36b85e546b5379bed12089ffd1bce01fcf
Co-Authored-By: Thiago da Silva <thiago@redhat.com>

F812 list comprehension redefines <variable> from line ...
While the current violations were benign, this sort of code can easily
lead to subtle bugs. Seems worth checking, especially given how cheap it
is to bring existing code in line with it.
Change-Id: Ibdcf9f93b85a1f1411198001df6bdbfa8f92d114

Change-Id: Ia78fbb32aa6f45aa6578734dc43f3262e717bab8

This commit updates the Swift Bandit file to the new style
introduced in Bandit 1.0. In response to the struggle with
getting a Bandit config file working and kept up to date we
introduced a simplified version in Bandit 1.0.
This commit updates Swift's bandit.yaml to use the new version.
Change-Id: Ida5dd08f4ea72a377346f2159caeb2f3741d4980

This patch improves EC GET response handling:
- The proxy no longer requires all object servers to have a
 durable file for the fragment archive that they return in
 response to a GET. The proxy will now be satisfied if just
 one object server has a durable file at the same timestamp
 as fragments from other object servers.
 This means that the proxy can now successfully GET an
 object that had missing durable files when it was PUT.
- The proxy will now ensure that it has a quorum of *unique*
 fragment indexes from object servers before considering a
 GET to be successful.
- The proxy is now able to fetch multiple fragment archives
 having different indexes from the same node. This enables
 the proxy to successfully GET an object that has some
 fragments that have landed on the same node, for example
 after a rebalance.
This new behavior is facilitated by an exchange of new
headers on a GET request and response between the proxy and
object servers.
An object server now includes with a GET (or HEAD) response:
- X-Backend-Fragments: the value of this describes all
 fragment archive indexes that the server has for the
 object by encoding a map of the form: timestamp -> <list
 of fragment indexes>
- X-Backend-Durable-Timestamp: the value of this is the
 internal form of the timestamp of the newest durable file
 that was found, if any.
- X-Backend-Data-Timestamp: the value of this is the
 internal form of the timestamp of the data file that was
 used to construct the diskfile.
A proxy server now includes with a GET request:
- X-Backend-Fragment-Preferences: the value of this
 describes the proxy's current preference with respect to
 those fragments that it would have object servers
 return. It encodes a list of timestamp, and for each
 timestamp a list of fragment indexes that the proxy does
 NOT require (because it already has them).
 The presence of a X-Backend-Fragment-Preferences header
 (even one with an empty list as its value) will cause the
 object server to search for the most appropriate fragment
 to return, disregarding the existence or not of any
 durable file. The object server assumes that the proxy
 knows best.
Closes-Bug: 1469094
Closes-Bug: 1484598
Change-Id: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

This patch update the outdated link reference inside the doc.
Change-Id: I9c5aabe869226489f361ff637e9dec731bbc3d7d

Specifically to facilitate the reuse of the retry check server
function to fill in the creds for the test2 account which is required
for probetests after the related change.
Change-Id: I9729faa4c8c8d6d65a481bc2ea3f0566d511034c
Related-Change: I8d503419b7996721a671ed6b2795224775a7d8c6

Currently the container sync daemon fails to copy
an SLO manifest, and the error will stall progress
of the sync process on that container. There are
several reasons why the sync of an SLO manifest
may fail:
1. The GET of the manifest from the source
 container returns an X-Static-Large-Object header
 that is not allowed to be included with a PUT
 to the destination container.
2. The format of the manifest object that is read
 from the source is not in the syntax required
 for a SLO manifest PUT.
3. Assuming 2 were fixed, the PUT of the manifest
 includes an ETag header which will not match the
 md5 of the manifest generated by the receiving
 proxy's SLO middleware.
4. If the manifest is being synced to a different
 account and/or cluster, then the SLO segments may
 not have been synced and so the validation of the
 PUT manifest will fail.
This patch addresses all of these obstacles by
enabling the destination container-sync middleware to
cause the SLO middleware to be bypassed by setting a
swift.slo_override flag in the request environ. This
flag is only set for request that have been validated
as originating from a container sync peer.
This is justifed by noting that a SLO manifest PUT from
a container sync peer can be assumed to have valid syntax
because it was already been validated when written to
the source container.
Furthermore, we must allow SLO manifests to be synced
without requiring the semantic of their content to be
re-validated because we have no way to enforce or check
that segments have been synced prior to the manifest, nor
to check that the semantic of the manifest is still valid
at the source.
This does mean that GETs to synced SLO manifests may fail
if segments have not been synced. This is however
consistent with the expectation for synced DLO manifests
and indeed for the source SLO manifest if segments have
been deleted since it was written.
Co-Authored-By: Oshrit Feder <oshritf@il.ibm.com>
Change-Id: I8d503419b7996721a671ed6b2795224775a7d8c6
Closes-Bug: #1605597 

Fixes a number of technical issues with the api-ref section
including:
- Added missing headers
- The header descriptions were made specific to whether they
 are used in requests or responses and the verb in question
 (example: Content-Length in object HEAD is the object size,
 not the response body length).
- Added references to API features such as bulk delete.
- Many typographical fixes (e.g., spaces in the middle
 of header names)
- Restored xml and json account/container listing
 examples.
The following areas were not updated and it is proposed
to defer them to a subsequent update. This is because
I don't have time or their merit is debatable:
- ACLs (as used in a Keystone context) are not
 described.
- Account create/delete is not described.
- I left List Endpoints as-is.
Change-Id: I315b4e550b7d10880fbc00fce9311127ba609c2d

Change-Id: I0d5c351735bdb6249dd9ec3e0c8b7ebe709feea6

With MAX_RANGES pulled out as a constant, developers are likely to
just trust the comment rather than go verify the value.
Change-Id: I47fbd4790492b272d53ee07fe5bc87170ce2c72b
Releated-Change: I4dcedcaae6c3deada06a0223479e611094d57234

There is a pop() function in the header_key_dict, but it has no
test_pop in the test_header_key_dict. Therefore, I add it.
Change-Id: I97bf9729fae3f044bcc69014549137f45ecbe91b

After test_shutdown,every method has its test case. Therefore, it is
not necessary to add more tests, then I remove TODO.
Change-Id: I23c1344bb18a83b154f890be0baa12799db045f8

Added man page for swift.conf
Change-Id: I88235ed414dfc34cb46f2fea014f1e721962885a
Closes-Bug: 1607027