openstack/swift - swift - OpenDev: Free Software Needs Free Tools

This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.
- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
 10.04 you'll have to pip install it, on 12.04 you can apt-get install
 it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.
Change-Id: I2e02684235aa2f4e901a00858ae037091594c545

fix some format according to pep8
Change-Id: I8ceb266d92c8997f131fa4096aaa6cfd4d585aaa

fir some format according to pep8
Change-Id: Iec8f67ea2166c955732bd598661140d4cfce6d40

Change-Id: Ie3423ce90d906948a1ce2db0efe3da184e60f6e0

Fix some pep8 issues in
 modified: test_bufferedhttp.py
 modified: test_constraints.py
 modified: test_db.py
 modified: test_db_replicator.py
 modified: test_init.py
make the code looks pretty.
Change-Id: I1c374b1ccd4f028c4e4b2e8194a6d1c201d50571

Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576

fallocate() failures properly return HTTPInsufficientStorage from
object-server before reading from wsgi.input, allowing the proxy
server to error_limit that node.
Change-Id: Idfc293bbab2cff1e508edf58045108ca1ef5cec1

Change-Id: Ie99250250171246e8c13e8d8c8258101bd78cce4

If rate_limit_after_segment was 10 in the proxy config, then after 10
segments were coughed up by _load_next_segment() with no calls to
sleep(), the 11th segment would not trigger a sleep() call. The 12th
segment triggered a sleep(0) call, but it was only after the 13th
segment was loaded that an actual rate-limiting (non-zero) sleep got
called.
With this patch, a rate_limit_after_segment of 10 will start sleeping
the correct amount after the 11th segment.
Updated proxy-server.conf-sample with rate_limit_after_segment and
rate_limit_segments_per_sec.
Change-Id: I937c366996e6d6ab47c614d6db470e3be9657c07

This is a pretty mechanical change that moves proxy controller
classes to their own files.
Change-Id: Iabb7bc134b07193e3bcc041a5c6890a0b95c2b55

To tell when replication for a device has finished, it's important to
know when the replicator is removing objects. This was previously
handled for the object-replicator
(object-replicator.partition.delete.count.<device> and
object-replicator.partition.update.count.<device> metrics) but not the
account and container replicators.
This patch extends the existing DB removal count metrics to make them
per-device. The new metrics are:
 account-replicator.removes.<device>
 container-replicator.removes.<device>
There's also a bonus refactoring and increased test coverage of the DB
replicator code.
Change-Id: I2067317d4a5f8ad2a496834147954bdcdfc541c1

Basically, do all hashing in the replicator without a lock, then
lock briefly to rewrite the hashes file. Retry if someone else has
modified the hashes file in the mean time (which should be rare).
Also, a little refactoring.
Change-Id: I6257a53808d14b567bde70d2d18a9c58cb1e415a

As we use the 'account_autocreate' for auto-create accounts, we send a
request to HEAD an account which is just deleted(the account-reaper not
reclaim it immediately), the proxy-server will return 'HTTP 500 Internal
Server Error'.
In my opinion, this is unreasonable, so I change the code in swift/proxy
/server.py for returning a reasonable response. I modified the code in
POST, GETorHEAD and account_info. At last, I modified some code in the
unittest test/unit/proxy/test_server.py.
Change-Id: Ib057b387c9da073d707ffae49ead54206a8fb7dd
Fixes: Bug #1037889 

Fixed bug where expirer would get confused by odd deletion times.
Since this has already rolled out, I just capped things at ten 9s, or
Sat Nov 20 17:46:39 2286. I can't wait for the Y2286 world panic. :/
Change-Id: Iba10963faa344a418a1fa573d5c85f4ff864b574

Serialize RingData in a versioned, custom format which is a combination
of a JSON-encoded header and .tostring() dumps of the
replica2part2dev_id arrays. This format deserializes hundreds of times
faster than rings serialized with Python 2.7's pickle (a significant
performance regression for ring loading between Python 2.6 and Python
2.7). Fixes bug 1031954.
swift.common.ring.ring.RingData is now responsible for serialization and
deserialization of its data via a new load() class method and save()
object method. The new implementation is backward-compatible; if a ring
does not begin with a new-style magic string, it is assumed to be an
old-style pickle-dumped ring and is handled as before. So new Swift
code can read old rings, but old Swift code will not be able to read
newly-serialized rings. THIS SHOULD BE MENTIONED PROMINENTLY IN THE
RELEASE NOTES.
I didn't want to bite of more than necessary, so I didn't mess with
builder file serialization.
Change-Id: I799b9a4c894d54fb16592443904ac055b2638e2d

We don't want to use pickle as it can execute arbitrary code. JSON is
safer. However, note that it supports serialization for only some
specific subset of object types; this should be enough for what we need,
though.
To avoid issues on upgrades (unability to read pickled values, and cache
poisoning for old servers not understanding JSON), we add a
memcache_serialization_support configuration option, with the following
values:
 0 = older, insecure pickle serialization
 1 = json serialization but pickles can still be read (still insecure)
 2 = json serialization only (secure and the default)
To avoid an instant full cache flush, existing installations should
upgrade with 0, then set to 1 and reload, then after some time (24
hours) set to 2 and reload. Support for 0 and 1 will be removed in
future versions.
Part of bug 1006414.
Change-Id: Id7d6d547b103b4f23ebf5be98b88f09ec6027ce4

It wasn't logging the full path, depending on if WebOb's
path_info_pop had been called.
Change-Id: I612d4cae7a4dc1bf5164356af329496245ad85dc

The WebOb 1.2 support patch introduced a regression where an unknown
format= query used to just default back to plain but now returns a
400. This will break anyone that uses format=text for instance, or
anything else "invalid".
Arguably, it makes the most sense to 400 in the case of an invalid
format requested, but since it is also a backwards compatibility
break, I recommend we keep the previous behavior. Retaining the
previous behavior seems less damaging than enforcing the new
behavior.
Change-Id: I6db015b33a6f3ab239b8cb4a3562ebdba1f76590

A warning log line is emitted whenever the proxy has to use a handoff
node. Monitoring these warnings can indicate a problem within your
cluster; however, you can disable these log lines by setting the
proxy conf's log_handoffs to false.
While working on this, I also noticed why many proxy log lines did
not have txn_id and client_ip -- subcoroutines. Now the logger thread
locals are copied to the subcoroutines.
Change-Id: Ibac086e1b985f566c068d083620287509de35da8

Change-Id: I7e5e254660e2ce35bc7fa95435db0a987d5acd0a

Change-Id: If5a6d32c40de02183a2eed6e2a32d62ba38df32d

In comments from https://review.openstack.org/8798 it was raised that it
might make sense to forbid some substrings in the name_filter
middleware.
There is now a new forbidden_regexp option for the name_filter
middleware to specify which substrings to forbid. The default is
"/\./|/\.\./|/\.$|/\.\.$" (or in a non-regexp language: the /./ and /../
substrings as well as strings ending with /. or /..).
This can be useful for extra paranoia to avoid directory traversals
(bug 1005908), or for more general filtering.
Change-Id: I39bf2de45b9dc7d3ca4d350d24b3f2276e958a62
DocImpact: new forbidden_regexp option for the name_filter middleware

don't check that number of servers equal 4.
Change-Id: I4a597eaf2b16b5b2cfc7fd706ac024357e9af2c2

Based on PatchSet 3 of https://review.openstack.org/#/c/7569/ , make them to pass all funcional tests with both webob 1.x and 1.2.
The additional following compatibility issues were addressed:
 - Until patch for range header issue is merged into official webob release, testRangedGetsWithLWSinHeader() should skip test against webob 1.2
(49c175aec2)
 - common.constraints.check_utf8() can accept both utf8 str and unicode.
 - To convert unicode to utf-8 str if necessary.
 - Making proxy_logging can handle invalid utf-8 str
bug 888371
bug 959881
blueprint webob-support
Change-Id: I00e5fd04cd1653259606a4ffdd4926db3c84c496

The probe tests were woefully out of date with all the changes that
have ocurred since they were written. I've updated most of them and
removed some that are hopeless outdated.
I also greatly improved the timing issues (hopefully completely
solved them? I ran them 25 times with no problems) and made them pep8
1.3.1 safe.
Change-Id: I8e9dbb6e7d6e04e293843b1dce1ded99d84e0348

- Rename it to keystoneauth for consistenties.
- Implements blueprint keystone-middleware.
Change-Id: I208fecdf3ee991694b4239f065032324d297fd35

swift.common.utils.validate_device_partition is a new function to check
that a device and a partition are valid. This means that they don't
contain '/' and are not '.' or '..'.
We use this new function every time we get devices and partitions from a
request.
Fix bug 1005908
Change-Id: Ia545ba8f877e85b4b576d6d7d09d890877ea6d34

In the interest of getting us to pep8 1.3.1 compliance I also fixed
all violations in the two files I was modifying anyway.
Fixes bug 1019051
Change-Id: I52eb7d59d2e2810a8cee2461f7ca869124e734e7

The X-Object-Manifest header value should be url encoded to allow for
"special" characters to be used. This fix decodes such encoding. A
corresponding fix to python-swiftclient (and any other clients) needs
to be made to url encode the header value when sending.
I also fixed any pep8 1.3.1 violations I found in the test I modified
to verify this.
Change-Id: I67f23dec1dc95ee37354dfdd9897f34758135ee6

Fixed bug 969411
Ranged GET request checked wheter the object is large objects only when response is 416 Requested Range Not Satisfiable.
All ranged GET requests should check the object is large object or not because size of the manifest file can be larger than 0.
Change-Id: Ia3632252d6879a773be881149c10ce0f5ffbcd82

Updated the imports and added a head_account to the "is the cluster
started yet?" checks. Hopefully this fixes the notorious timing
issues of these tests where auth answers requests just a bit before
the rest of the cluster is ready.
Fixes bug 1014931
Change-Id: Iea1d62db2317560371da49af5e94a0279b646294

The old use of Eventlet's posthook process meant that responses that
forgot to include content-length or transfer-encoding headers would
get one tacked on, if Eventlet could guess what was probably meant. I
added a bit of that logic into proxy-logging now as we saw some
errors resulting from this.
Fixes Bug #1012714
Change-Id: I671453eaf3704eab814ff12c4625ba7d749cc7ed

bug 786197
Change-Id: Idd73986ca4550b20dff340df4acd12f3bfc01075