openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Adds section to detail how to prevent disk full scenarios from
occurring.
Change-Id: Iafb4a47fa4892f6067252f3a80de87cd76506a40

swiftsync project is no longer maintained and so
is removed from the list
Change-Id: Ieae67c728da16e5516babd33d2e5937b14738d6a
Closes-Bug: #1581975 

The testPUT case is failing when keystone was enabled and
allow_account_management is set to True. There were a few issues
needing addressed.
First the case was renamed to call out what it was actually doing which
is verifying an error scenario for which a PUT on a storage account was
not allowed.
Second the case was running even when allow_account_management is
enabled, which is incorrect. It "accidently" works with TempAuth
because it requires a reseller permission, so the Keystone failure here
has more to do with not requiring a reseller permission to do a PUT on a
storage account for which a user has an operator role on.
The common sense fix here is to not execute this test case when
allow_account_management is enabled.
Change-Id: Id29f5ca48f92cd139535be7064107b8a61b02856

get_account_info used to work like this:
 * make an account HEAD request
 * ignore the response
 * get the account info by digging around in the request environment,
 where it had been deposited by elves or something
Not actually elves, but the proxy's GETorHEAD_base method would take
the HEAD response and cache it in the response environment, which was
the same object as the request environment, thus enabling
get_account_info to find it.
This was extraordinarily brittle. If a WSGI middleware were to
shallow-copy the request environment, then any middlewares to its left
could not use get_account_info, as the left middleware's request
environment would no longer be identical to the response environment
down in GETorHEAD_base.
Now, get_account_info works like this:
 * make an account HEAD request.
 * if the account info is in the request environment, return it. This
 is an optimization to avoid a double-set in memcached.
 * else, compute the account info from the response headers, store it
 in caches, and return it.
This is much easier to think about; get_account_info can get and cache
account info all on its own; the cache check and cache set are right
next to each other.
All the above is true for get_container_info as well.
get_info() is still around, but it's just a shim. It was trying to
unify get_account_info and get_container_info to exploit the
commonalities, but the number of times that "if container:" showed up
in get_info and its helpers really indicated that something was
wrong. I'd rather have two functions with some duplication than one
function with no duplication but a bunch of "if container:" branches.
Other things of note:
 * a HEAD request to a deleted account returns 410, but
 get_account_info would return 404 since the 410 came from the
 account controller *after* GETorHEAD_base ran. Now
 get_account_info returns 410 as well.
 * cache validity period (recheck_account_existence and
 recheck_container_existence) is now communicated to
 get_account_info via an X-Backend header. This way,
 get_account_info doesn't need a reference to the
 swift.proxy.server.Application object.
 * both logged swift_source values are now correct for
 get_container_info calls; before, on a cold cache,
 get_container_info would call get_account_info but not pass along
 swift_source, resulting in get_account_info logging "GET_INFO" as
 the source. Amusingly, there was a unit test asserting this bogus
 behavior.
 * callers that modify the return value of get_account_info or of
 get_container_info don't modify what's stored in swift.infocache.
 * get_account_info on an account that *can* be autocreated but has
 not been will return a 200, same as a HEAD request. The old
 behavior was a 404 from get_account_info but a 200 from
 HEAD. Callers can tell the difference by looking at
 info['account_really_exists'] if they need to know the difference
 (there is one call site that needs to know, in container
 PUT). Note: this is for all accounts when the proxy's
 "account_autocreate" setting is on.
Change-Id: I5167714025ec7237f7e6dd4759c2c6eb959b3fca

Previously, if you called get_account_info, get_container_info, or
get_object_info, then the results of that call would be cached in the
WSGI environment as top-level keys. This is okay, except that if you,
in middleware, copy the WSGI environment and then make a subrequest
using the copy, information retrieved in the subrequest is cached
only in the copy and not in the original. This can mean lots of extra
trips to memcache for, say, SLO validation where the segments are in
another container; the object HEAD ends up getting container info for
the segment container, but then the next object HEAD gets it again.
This commit moves the cache for get_*_info into a dictionary at
environ['swift.infocache']; this way, you can shallow-copy the request
environment and still get the benefits from the cache.
Change-Id: I3481b38b41c33cd1e39e19baab56193c5f9bf6ac

The probe test in question tries to make a manifest referencing
unavailable objects and expects that to fail with a 400. If the SLO
middleware is enabled, then it checks the segments, can't find some,
and returns the 400. If it's disabled, the PUT succeeds and makes an
object whose contents are some JSON blob. In the latter case, the
probe test would fail because it expected a real SLO manifest but
didn't find one.
Now we skip the remainder of the test when we detect that SLO is not
enabled.
Change-Id: I3e7e8e98107608e675efc24156e703bc167458bb

Change-Id: Ib3cfe38d1b17c09c7248b5cbcfe4e2eadff7acd9

Change-Id: I2b6ca3f9277b344ff5b05a85304084a7edcbcb66

Change-Id: I01f1ad8ef0f8910718fd2fb30c9e8285358baf84

We lost some unit tests from test_sysmeta.py in [1].
This patch restores them. Since the proxy no longer
implements post-as-copy or COPY then we need to plumb
in a copy middleware to the test app in test_sysmeta.
It is a little odd perhaps to have these tests under
test/unit/proxy but the alternative is to duplicate
the test setup and base test code that already exists
in test_sysmeta.py into test_copy.py.
[1] commit 46d61a4dcd
 (server side copy middleware)
Change-Id: Iec02387ccbddbe3841a417880389c707cd5c0346

There is a broken link to CONTRIBUTING.rst
introduced in commit a829bd5977
Change-Id: Iaedf5ff3995229cf292202793809080f9f2c7fed

If one uses only a single replica and a database file is placed on a
wrong partition, it will be removed instead of replicated to the correct
partition.
There are two reasons for this:
1. The list of nodes is empty when there is only a single replica
2. all(responses) is True even if there is no response at all, and the
latter is always True if there is no node to replicate to.
This patch fixes this by adding a special case if used with only one
replica to the node selection loop and ensures that the list of
responses is not empty. Also adds a test that fails on current master
and passes with this change.
Closes-Bug: 1568591
Change-Id: I028ea8c1928e8c9a401db31fb266ff82606f8371

Add an other-requirements.txt file containing a cross-platform list of
dependencies needed for running included tox-based tests. Also include a
tox environment for convenience calling the bindep[*] utility to list
any missing system requirements.
Document bindep and other-requirements usage.
This change is self-testing.
For other-requirements.txt see also
http://docs.openstack.org/infra/manual/drivers.html#package-requirements
[*] http://docs.openstack.org/infra/bindep/
Change-Id: Iea6f5fecba3b7cb9f6dac7029c0f17fc31fc0e3c

Following fd86d5a, the object-auditor would leave status files so it
could resume where it left off if restarted. However, this would also
cause the object-reconstructor to print warnings like:
 Unexpected entity in data dir: u'/srv/node4/sdb8/objects/auditor_status_ZBF.json'
...which isn't actually terribly useful or actionable. The auditor will
clean it up (eventually); the operator doesn't have to do anything.
Now, the reconstructor will specifically ignore those status files.
Change-Id: I2f3d0bd2f1e242db6eb263c7755f1363d1430048

Rewrite server side copy and 'object post as copy' feature as middleware to
simplify the PUT method in the object controller code. COPY is no longer
a verb implemented as public method in Proxy application.
The server side copy middleware is inserted to the left of dlo, slo and
versioned_writes middlewares in the proxy server pipeline. As a result,
dlo and slo copy_hooks are no longer required. SLO manifests are now
validated when copied so when copying a manifest to another account the
referenced segments must be readable in that account for the manifest
copy to succeed (previously this validation was not made, meaning the
manifest was copied but could be unusable if the segments were not
readable).
With this change, there should be no change in functionality or existing
behavior. This is asserted with (almost) no changes required to existing
functional tests.
Some notes (for operators):
* Middleware required to be auto-inserted before slo and dlo and
 versioned_writes
* Turning off server side copy is not configurable.
* object_post_as_copy is no longer a configurable option of proxy server
 but of this middleware. However, for smooth upgrade, config option set
 in proxy server app is also read.
DocImpact: Introducing server side copy as middleware
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: Ic96a92e938589a2f6add35a40741fd062f1c29eb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Signed-off-by: Thiago da Silva <thiago@redhat.com>

These are blog posts authored by Greg Holt (gholt) and used with
permission to add here. Content was only reformatted as rst and
wrap prose lines at 70 characters.
Change-Id: I7aa47c24b5019aa598ee005e01612a49514da25f

hash_cleanup_listdir was removed in [1], this patch
renames all references to it in test_diskfile to refer to
the cleanup_ondisk_files method that is now tested directly.
Also remove the final references to the now non-existent
function in a few comments.
[1] I0b96dfde32b4c666eebda6e88228516dd693ef92
Change-Id: I1e151799fc2774de9a1af092afff875af24a630c
Related-Bug: #1550569 

This started as a new "new_contributor" doc. But we've already got
at least 3 different docs like that.
Change-Id: Ia2303ab55eeea01cc71acbccaeab55dad0ef5ff9

This has been in eventlet.listen() since version 0.15.
Change-Id: Ib77b28231a2180f1ea082f356c4687c39681a6f7

Before this commit, when a local device has not found been found
in a object-replication run, the policy was not mentioned in the
error log. But it is of interest to know the policy, for example for
error searching, when no local device has been found.
Change-Id: Icb9f9f1d4aec5c4a70dd8abdf5483d4816720418

Commit cf48e75 changed the default account/container/object ports in a
lot of places, including the probetests. However, it didn't change
them in doc/saio/bin/remakerings, and since the probe tests must match
the rings, they started failing.
This commit just backs out the changes to the test/probe directory so
that remakerings and the probe tests match again.
Change-Id: I316a09e6ee1a911f37ce9df3d641644739f88eeb

Changing the recommended ports for Swift services
from ports 6000-6002 to unused ports 6200-6202;
so they do not conflict with X-Windows or other services.
Updated SAIO docs.
DocImpact
Closes-Bug: #1521339
Change-Id: Ie1c778b159792c8e259e2a54cb86051686ac9d18

With the removement of threads_per_disk there is no longer a need to use
run_in_thread() at all; it was just calling the function itself when
running with 0 threads.
Similar to force_run_in_thread() - with 0 threads it was basically doing
the same like in tpool_reraise(), therefore replacing the call and
finally removing the complete ThreadPool class.
Note that this might break external consumers that are inheriting
BaseDiskFileManager; in this case you need to adopt this change in your
codebase then.
Change-Id: I39489dd660935bdbfbc26b92af86814369369fb5

Previously SwiftLogFormatter would make two checks. One to see if the
transaction id was already in the message field and another check to
make sure the log level wasn't set to info. If either of these was
true, then it would not log the transaction ID in the transaction ID
field.
This commit removes the check for the info log. Now transaction IDs
will be recorded in all cases that have them.
Change-Id: Ic06538ab55a75d298169ae1745671573ee9c09e8
Closes-Bug: #1504344