openstack/swift - swift - OpenDev: Free Software Needs Free Tools

My author name is my gmail account name, change it to my real name.
Change-Id: Iafd94f694f7d007c54de086941309b119683f6cb

Change-Id: I2d147415e65c0f19159b995fc4eca78e075537b6

Change-Id: I457674f564dcacf0fd20004e9a3bb85113f53aa5

Prior to this commit, swift-ring-builder would place partitions on
devices by first going for maximal dispersion and breaking ties with
device weight. This commit flips the order so that device weight
trumps dispersion.
Note: if your ring can be balanced, you won't see a behavior
change. It's only when device weights and maximal-dispersion come into
conflict that this commit changes anything.
Example: a cluster with two regions. Region 1 has a combined weight of
1000, while region 2 has a combined weight of only 400. The ring has 3
replicas and 2^16 partitions.
Prior to this commit, the balance would look like so:
 Region 1: 2 * 2^16 partitions
 Region 2: 2^16 partitions
After this commit, the balance will be:
 Region 1: 10/14 * 2^16 partitions (more than before)
 Region 2: 4/14 * 2^16 partitions (fewer than before)
One consequence of this is that some partitions will not have a
replica in region 2, since it's not big enough to hold all of them.
This way, a cluster operator can add a new region to a single-region
cluster in a gradual fashion so as not to destroy their WAN link with
replication traffic. As device weights are increased in the second
region, more replicas will shift over to it. Once its weight is half
that of the first region's, every partition will have a replica there.
DocImpact
Change-Id: I945abcc4a2917bb12be554b640f7507dd23cd0da

Previously, any new (partition, replica) pairs created by adding
replicas were spread evenly across the whole ring. Now it respects
device weights, so more (partition, replica) pairs are placed on
higher-weight devices.
Note that this only affects partition assignment *when the replica
count is increased*. Normal adding of disks without changing the
replica count is, and was, just fine.
The cause was that the devices' parts_wanted values weren't being
updated when the replica count went up. Since adding replicas makes
more things to assign, each device's desired share should have gone
up, but it didn't. This appears to be a simple oversight on the part
of the original author (me).
Change-Id: Idab14be90fab243c1077a584396a9981a4bd8638

Same fix as is going through on feature/ec, fixed it there first
as it was happening often and no point in waiting until EC is
complete before getting it over to master...
Appears that what's been happening on feature/ec lately
with a middleware failure has to do with a hardcoded PID in
the test code itself causing a profile file to exist when
its not expected to by the test.
Test code used a PID of 135 and based on how get_logfiles()
is written, any real PID that starts with 135 will cause a
false failure in test_call(). This can be seen via inspection
and confirmed in logfiles where all captured assertions show
a profile filename beginning with 135. Tried getting smarter
about choosing a fake PID (int) but then decided it was 100%
safe to use 'ABC' for this test since that'll never show up!
Change-Id: I958f1525c2727b3fb3f533242fa509fa8e59926c

added account_autocreate and allow_account_management to
the /info endpoint
Change-Id: I4b239c9cefb728c3c93bf75cad065c72edf2fc0a

All the expiring objects for a given X-Delete-At are funnelled into the
same expiring object container- this can act as a bottleneck.
Change-Id: I288a177a7ae3e213c727a2a81fa76d4ef9cf7eb3

When lock_path is called and the lock goes for the whole 10 seconds,
the flock is called 1000 times. With this patch, the short 0.01 sleep
is used for the first 1% of the total lock time and then 1% of the
total lock time is used.
Change-Id: Ibed6bdb49bddcdb868742c41f86d2482a7edfd29

Closes-Bug: #1183152
Change-Id: I4b2c6e947241c987779a385fdff270d037470a57

Account-reaper works only at account-server with the first replica, and reaps
account with "deleted" status.
On the other hand, account-replicator doesn't replicate the status, only
replicates *_timestamp.
When swift fails to delete the first account replica, account-reaper never
reaps the account, because the first replica never gets marked as "deleted".
This patch adds a timestamp checking into is_status_deleted method, and
account-reaper will start to reap the account after account-replicator
replicates *_timestamp.
Change-Id: I75e3f15ad217a71b4fd39552cf6db2957597efca
Closes-Bug: #1304755 

There was some duplication of code in both POST and PUT
methods to handle object-expiration headers.
A method was created to remove this duplication,
which should help with maintainability of code.
Change-Id: I85cc4a7b0d688760c97598d80b9e9a39288c5f34
Signed-off-by: Thiago da Silva <thiago@redhat.com>

Adds a description of the methods config option to the
docstring.
Also fixes description of swift.source value and a
couple of other typos.
Change-Id: If3489087df020536ff663ffe4e249c300ea2d506

The tempurl middleware supports any configured HTTP methods, but the
default set was only GET, PUT, and HEAD, so cluster operators had to
take action to enable POST and DELETE. This commit changes the
defaults to include POST and DELETE.
Note that this doesn't affect any existing temporary URLs at all; the
method is baked into the signature (temp_url_sig query param), so no
new access is granted to a holder of a temporary URL by this
change. It simply gives more flexibility to creators of temporary
URLs.
Change-Id: I5bc15bbd2968ab7bedcd7c0df10f2ec825537191

Config items 'rate_limit_after_segment' and
'rate_limit_segments_per_sec' are configuration items left over from
before DLO was moved from the proxy server and into it's own
middleware. This change removes the configuration options from the
proxy server.
Change-Id: Ie3d42a57cda813feed019f01db986f13cdff9c31

This change is the result of an audit through the config parameters
provided by swift and how/if they are addressed in the swift
documentation. The documentation being the sample config files in
the /etc directory or the documentation.
This change is only concerned with the config files in etc/ next
I will look at the documentation in the doc/ folder.
This change makes the following assumptions:
 - Unless stated otherwise, the commented out parameter in the
 sample configuration is the default for swift.
 - When the default in the code differs from that of the sample
 configuration, the default in the code is correct.
Container reconciler:
 Parameter: interval
 - code: 30
 - config: 300
 Result: config = 30
Object Expirer:
 Parameter: recon_cache_path
 - code: /var/cache/swift
 - config: Parameter missing
 Result: Add parameter
swift-dispersion-populate && swift-dispersion-report
 Parameter: auth_version
 - code: 1.0
 - config: 2.0 (due to being a confusing example of how to setup
 version 2.0).
 Result: Added 'auth_version = 1.0' to the right section (showing
 default and make the sample configuration for auth version
 2.0 easier to understand.
swift-drive-audit:
 Parameter: log_file_pattern
 - code: /var/log/kern.*[!.][!g][!z]
 - config: /var/log/kern*
 Result: config = /var/log/kern.*[!.][!g][!z]
 NOTE: swift-drive-audit uses a parameter called device_dir which
 defaults to '/srv/node'. In all other swift binaries/services
 there is a similar parameter called devices which stores the
 same thing. This is an inconsistency which I haven't fixed
 as this could break existing swift clusters out in the wild.
Proxy Server:
 Parameter: object_chunk_size
 - code: 65536
 - config: 8192
 Result: config = 65536
 Parameter: client_chunk_size
 - code: 65536
 - config: 8192
 Result: config = 65536
 Parameter: strict_cors_mode
 - code: True
 - config: No parameter
 Result: config = True
Account and Container replicator configuration confusion:
 NOTES:
 The account and container replicators have parameters:
 - interval
 - run_pause
 Both of these are loaded into the same variable in code:
 self.interval = int(conf.get('interval') or
 conf.get('run_pause') or 30)
 If a user sets both to different values then interval is used.
 Result: Update the configuration to make this more clear.
DocImpact
Change-Id: Iaadbb1a6284f8b3e0801bc343b29772f70f4bf6e

swift-form-signature would give you the required expiration-time and
HMAC signature, but it wouldn't help you actually construct the HTML
form. To do that, you had to go look at the formpost middleware's doc
string and make up a form yourself.
For convenience, this commit makes swift-form-signature output a
sample form with the computed values filled in already; the user only
needs to fill in the Swift cluster's hostname.
Change-Id: I70d70a648b78b382dbfbe8ff918e6158a7f6a0ab

Currently the theme used by the swift developer docs are out of
date, it should be using oslosphinx, to provide a similar look
and feel between all openstack related projects.
Change-Id: Id7c226cdc13c6c4f3b5082b1ef4dfe09966b21ec

This function was calling a couple functions prior to checking for
early bailout; if it did bail out, the results of those function calls
were unused. Bail out earlier to avoid wasted work.
Change-Id: Ia023549a45355b44ebfb41212565c42f993415a9

Add an environment variable to enable the use of the in-memory object
server during in-process functional test runs.
It might be worth-while to just run under both object servers in-tree,
but this at least enables it, without having to figure out how to make
two test runs in two different environments.
DocImpact
Change-Id: Id76b008e1f273c639ae61550affddc32c5d7c419
Signed-off-by: Thiago da Silva <thiago@redhat.com>

If the container names in the expirer's account are returned as
unicode strings (as is the case with some json libraries), the
expirer compared eg u'1' == '1', which is problematic. This patch
ensures that the unicode is coerced to ascii so the comparison
is correct.
Change-Id: I72b322e7513f7da32e8dc75c6bf0e7e016948c88

This patch takes a first step towards support
for object system metadata by enabling headers
in the x-object-sysmeta- namespace to be
persisted when objects are PUT. This should be
useful for other pending patches such as on
demand migration and server side encryption
(https://review.openstack.org/#/c/64430/ and
https://review.openstack.org/#/c/76578/1).
The x-object-sysmeta- namespace is already
reserved/protected by the gatekeeper and
passed through the proxy. This patch modifies
the object server to persist these headers
alongside user metadata when an object is
PUT.
This patch will preserve existing object
system metadata and ignore any new system
metadata when handling object POSTs,
including POST-as-copy operations. Support
for modification of object system metadata
with a POST request requires further work
as discussed in the blueprint.
This patch will preserve existing object
system metadata and update it with new
system metadata when copying an object.
A new probe test is added which makes use of
the BrainSplitter class that has been moved
from test_container_merge_policy_index.py to
a new module brain.py.
blueprint object-system-metadata
Change-Id: If716bc15730b7322266ebff4ab8dd31e78e4b962