openstack/swift - swift - OpenDev: Free Software Needs Free Tools

swift/container/sync.py uses swift.common.internal_client.delete_object
and put_object and expected these methods raise ClientException.
But delete_object and put_object never raise the exception so this patch
raises ClientException when urllib2 library raises HTTPError.
Co-Authored-By: Eran Rom <eranr@il.ibm.com>
Closes-Bug: #1419901
Change-Id: I58cbf77988979a07998a46d9d81be84d29b0d9bf

Change-Id: I4764ba505646949ff694f8939947f83c6940128a

It's harder than it sounds. There was really three challenges.
Challenge #1 Initial Assignment
===============================
Before starting to assign parts on this new shiny ring you've
constructed, maybe we'll pause for a moment up front and consider the
lay of the land. This process is called the replica_plan.
The replica_plan approach is separating part assignment failures into
two modes:
 1) we considered the cluster topology and it's weights and came up with
 the wrong plan
 2) we failed to execute on the plan
I failed at both parts plenty of times before I got it this close. I'm
sure a counter example still exists, but when we find it the new helper
methods will let us reason about where things went wrong.
Challenge #2 Fixing Placement
=============================
With a sound plan in hand, it's much easier to fail to execute on it the
less material you have to execute with - so we gather up as many parts
as we can - as long as we think we can find them a better home.
Picking the right parts for gather is a black art - when you notice a
balance is slow it's because it's spending so much time iterating over
replica2part2dev trying to decide just the right parts to gather.
The replica plan can help at least in the gross dispersion collection to
gather up the worst offenders first before considering balance. I think
trying to avoid picking up parts that are stuck to the tier before
falling into a forced grab on anything over parts_wanted helps with
stability generally - but depending on where the parts_wanted are in
relation to the full devices it's pretty easy pick up something that'll
end up really close to where it started.
I tried to break the gather methods into smaller pieces so it looked
like I knew what I was doing.
Going with a MAXIMUM gather iteration instead of balance (which doesn't
reflect the replica_plan) doesn't seem to be costing me anything - most
of the time the exit condition is either solved or all the parts overly
aggressively locked up on min_part_hours. So far, it mostly seemds if
the thing is going to balance this round it'll get it in the first
couple of shakes.
Challenge #3 Crazy replica2part2dev tables
==========================================
I think there's lots of ways "scars" can build up a ring which can
result in very particular replica2part2dev tables that are physically
difficult to dig out of. It's repairing these scars that will take
multiple rebalances to resolve.
... but at this point ...
... lacking a counter example ...
I've been able to close up all the edge cases I was able to find. It
may not be quick, but progress will be made.
Basically my strategy just required a better understanding of how
previous algorithms were able to *mostly* keep things moving by brute
forcing the whole mess with a bunch of randomness. Then when we detect
our "elegant" careful part selection isn't making progress - we can fall
back to same old tricks.
Validation
==========
We validate against duplicate part replica assignment after rebalance
and raise an ERROR if we detect more than one replica of a part assigned
to the same device.
In order to meet that requirement we have to have as many devices as
replicas, so attempting to rebalance with too few devices w/o changing
your replica_count is also an ERROR not a warning.
Random Thoughts
===============
As usual with rings, the test diff can be hard to reason about -
hopefully I've added enough comments to assure future me that these
assertions make sense.
Despite being a large rewrite of a lot of important code, the existing
code is known to have failed us. This change fixes a critical bug that's
trivial to reproduce in a critical component of the system.
There's probably a bunch of error messages and exit status stuff that's
not as helpful as it could be considering the new behaviors.
Change-Id: I1bbe7be38806fc1c8b9181a722933c18a6c76e05
Closes-Bug: #1452431 

... to match global-requirements. This is an
intermediate step in updating requirements to
PyECLib>=1.1.1.
Change-Id: I79f47fff6ec1adff214ca435f0d95aaf1ffd68f9

Currently fails on master, passes with change
If13af444ed301ebd8fd34a0d96a330ded648f0c4 applied.
Change-Id: I05b231cc232d5b4117bccee40eebc7093114c61c

Change-Id: I38a7bd7b85de9d674386d75220b31cc467746da8

Change-Id: I4b4fd9179d0234f001940e215c97d40a2a6204cd

Also, clean up the module documentation a bit.
Change-Id: Iaeb5eb264b118b78738187db9242540275e77444

Updated string prefix and suffix checker slicing to startswith()
and endswith() methods.
Using startswith() and endswith() improves readability, error-proneness
and enhances maintainability.
Change-Id: I1d5fbf116a61763346c6f92fd8023dbbe9bb37cf

Updates the functional test to verify the fix applied by
change Iff7274aa631a92cd7332212ed8b4378c27da4a1f
Change-Id: Iae63ac027e4f4acfe46a36dc1325888b1f834ea4

Change-Id: I9d417faede707e4f3570074e410344cc8955007b

If a SLO manifest specified the same segment twice with no range on
the latter, we would crash while trying to coalesce the segment
specifications into a single request (for efficiency). Now we handle
that missing range and do the right thing.
Change-Id: If13af444ed301ebd8fd34a0d96a330ded648f0c4

The content-type header inserted into a multipart message part
is missing any params such as charset because its value is being
fetched via the swob.Response content_type property, which conforms
to webob spec and strips off all params.
This was noticed in work on feature/crypto branch because the crypto
meta param was being stripped off content-type in multipart messages,
preventing the content-type being decrypted. But in general there is
no suggestion in the multipart message spec [1] that params should
not be included. In fact examples in [1] show the charset param
included in the content-type value.
To ensure that the multipart message part content-type includes the
original content-type params, fetch it directly from the response
headers.
[1] http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html
Change-Id: Iff7274aa631a92cd7332212ed8b4378c27da4a1f

Clarify the behavior of strict and non-strict options.
Define what <server> and <config> specify on the command line.
Drive-by fix for missing param in search_tree docstring.
Change-Id: I89fff88a088bc73001464b1fa8240e14a61a117d

This will reduce the overhead required for versioned containers in terms
of memory, back-end requests, and wall time.
Change-Id: I156216a14141db547de3cae1dcd5315ae31eaac2

Also fix typo in the api doc.
Follow-on to change I5eb655360ac95042877da26d18707aebc11c02f6
Change-Id: Ic1ea63c74a0a3b90a505865fba8fcfac584d0825

* With the end_prefix changes in the original commit, we no longer need
 the `or not name.startswith(prefix)` check.
 * Improve test coverage of reverse path listings.
Change-Id: Iaa7d4b83647c3c150be95f88cb3cc9e4f0e33979

This change adds the ability to tell the container or account server to
reverse their listings. This is done by sending a reverse=TRUE_VALUE,
Where TRUE_VALUE is one of the values true can be in common/utils:
 TRUE_VALUES = set(('true', '1', 'yes', 'on', 't', 'y'))
For example:
 curl -i -X GET -H "X-Auth-Token: $TOKEN" $STORAGE_URL/c/?reverse=on
I borrowed the swapping of the markers code from Kevin's old change,
thanks Kevin. And Tim Burke added some real nuggets of awesomeness.
DocImpact
Co-Authored-By: Kevin McDonald <kmcdonald@softlayer.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Implements: blueprint reverse-object-listing
Change-Id: I5eb655360ac95042877da26d18707aebc11c02f6

Add test for database create request without account
Partial test for migrate call on database with storage_policy_index
Change-Id: I7cfbd6bc7e2b341f433d88f600b19e54826a0e22
Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>

Fix/add entries for authors that are known to now have
@hpe.com email addresses.
Change-Id: I62c83b64e2378cb3d6ad33ac9e6ab111b8a8c86f

Example:
* Different port in config and in ring file.
* Running daemon on server not in ring file.
In both cases replication daemon is running but nothing is replicated.
Error log helps to distinguish a local device can't be identified.
Closes-Bug: 1508228
Change-Id: I99351b7d9946f250b7750df91c13d09352a145ce

Otherwise, if a client includes something like:
 Content-Type: application/x-www-form-urlencoded
...we won't delete anything, and instead send back:
 Number Deleted: 0
 Number Not Found: 0
 Response Body:
 Response Status: 406 Not Acceptable
 Errors:
...despite the fact that we're managing the iterator, so we know it's
acceptable.
Change-Id: I791c7bda1d9df830d8dacd3783c2393db5a9ac09

Added unit test cases to cover all code path in _one_audit_pass
function in container/auditor.py
Change-Id: I8b89e94f1492e4366af3ac4260587e988ba29408

Add a py34 test environment to tox.ini using a whitelist
of tests which pass on Python 3. Currently, only test_exceptions.py
of test/unit/common/ is run on Python 3
Change-Id: I5c8a9fe7b9635f1acab8f401908975fc6fc58c7a