openstack/swift - swift - OpenDev: Free Software Needs Free Tools

The probe tests were woefully out of date with all the changes that
have ocurred since they were written. I've updated most of them and
removed some that are hopeless outdated.
I also greatly improved the timing issues (hopefully completely
solved them? I ran them 25 times with no problems) and made them pep8
1.3.1 safe.
Change-Id: I8e9dbb6e7d6e04e293843b1dce1ded99d84e0348

- Rename it to keystoneauth for consistenties.
- Implements blueprint keystone-middleware.
Change-Id: I208fecdf3ee991694b4239f065032324d297fd35

swift.common.utils.validate_device_partition is a new function to check
that a device and a partition are valid. This means that they don't
contain '/' and are not '.' or '..'.
We use this new function every time we get devices and partitions from a
request.
Fix bug 1005908
Change-Id: Ia545ba8f877e85b4b576d6d7d09d890877ea6d34

In the interest of getting us to pep8 1.3.1 compliance I also fixed
all violations in the two files I was modifying anyway.
Fixes bug 1019051
Change-Id: I52eb7d59d2e2810a8cee2461f7ca869124e734e7

The X-Object-Manifest header value should be url encoded to allow for
"special" characters to be used. This fix decodes such encoding. A
corresponding fix to python-swiftclient (and any other clients) needs
to be made to url encode the header value when sending.
I also fixed any pep8 1.3.1 violations I found in the test I modified
to verify this.
Change-Id: I67f23dec1dc95ee37354dfdd9897f34758135ee6

Fixed bug 969411
Ranged GET request checked wheter the object is large objects only when response is 416 Requested Range Not Satisfiable.
All ranged GET requests should check the object is large object or not because size of the manifest file can be larger than 0.
Change-Id: Ia3632252d6879a773be881149c10ce0f5ffbcd82

The documentation could be understood like the following:
memcache_servers from memcache.conf is always used if set, even if
memcache_servers in proxy-server.conf is set.
This is clearly not the case, as proxy-server.conf has a higher priority
if it memcache_servers is set there.
Change-Id: I967c7e80796a0e296c5c65bd097df1669d16203e

- Fixes bug 898169.
- It's possible to make GzipFile inherits io.BufferedIOBase in python
 2.6, but IMO it's better let people to use 2.7 if they really need
 this feature.
Change-Id: I4c6e4858e3f616af892e7f4fa6daae0f6ee31723

When a syslog daemon fails and SysLogHandler cannot reconnect to
it, server processes (proxy, container, object, and more)
would start consuming 100 % CPU and stop responding to requests.
It is because the server process go into an infinite loop if they
fail to reconnect to /dev/log.
This problem happens only when using unix_sockets.
This change enables us to use syslog without unix_sockets and fixes bug 780025.
Change-Id: Ibcc99a1a148b1008036146bf3bd079a9be24982c

Updated the imports and added a head_account to the "is the cluster
started yet?" checks. Hopefully this fixes the notorious timing
issues of these tests where auth answers requests just a bit before
the rest of the cluster is ready.
Fixes bug 1014931
Change-Id: Iea1d62db2317560371da49af5e94a0279b646294

Fixes LP Bug #1016656.
Change-Id: I22455d27cb82830dabd4fb0677da26915a60eb85

Fixes bug 1016140.
Change-Id: I0c98561f58f80408064ea178c6d6e8889604cddb

Commit 95786e5 made swift-get-nodes only output N handoff nodes, where
N is the ring's replica count. This change leaves that behavior in
place as the default, but adds the option -a/--all to print all the
handoff nodes.
This is handy if you're consuming the output with a script, e.g. if
you're doing some analysis of the distribution of primary+handoff
nodes for a particular set of things.
Change-Id: Ie0253575bb21ccf25559a1c5fd20570a58a2d9fa

* Add new configuration option log_address.
Change-Id: I636bd4116687629c997b70a0d804b7ed4bc46032

This changes the JSON output keys missing_one and missing_two to
missing_1 and missing_2 (if there are at least 2 replicas), drops
the missing_all key, and can add more missing_x keys depending on
the replica count. We will definitely need to post a prominent
notice of this change with the next release.
bug 862816
Change-Id: Ib747d944476638c33ee1d876b8f9db28953826e2

If disk failure happens while object replication is ongoing,
object-replicator could replicate objects from other nodes to the root
file system. It is because object-replicator does no mount check
(error check) at all after collect_jobs() is called, and it executes
rsync based on the old partition information in update().
This change prevents many objects from being written to the root file
system wrongly and fixes bug 1014545.
Change-Id: Icf740a2683e360df89eccb71b269977f294340b9

The old use of Eventlet's posthook process meant that responses that
forgot to include content-length or transfer-encoding headers would
get one tacked on, if Eventlet could guess what was probably meant. I
added a bit of that logic into proxy-logging now as we saw some
errors resulting from this.
Fixes Bug #1012714
Change-Id: I671453eaf3704eab814ff12c4625ba7d749cc7ed

Fixes Bug #1011727
Change-Id: I6681bd85a5ddb82a1059913ae90696e5994aa906

Changed swift-get-nodes to just return up to replica_count of
more_nodes instead of all of them. When you have thousands of drives,
returning all possible handoff nodes is pretty crazy. :)
Change-Id: I864e7243726ca3ef2ac08c4ab660a05789f37878

The jenkins coverage jobs expect there to be a .coverage file, so deleting
it is a bad idea. Also, coverage erase will do that for us.
While we're in there, update tox.ini and setup.cfg to the latest.
Change-Id: Icd0a8fc66a5146e0d94f62a9f65a4536981d2916

The object server had a constant KEEP_CACHE_SIZE = 5*1024*1024;
unauthenticated GET requests for files smaller than KEEP_CACHE_SIZE
would not evict the file from the kernel's buffer cache after it was
read from disk.
Now that hardcoded constant is a configuration parameter
("keep_cache_size"), and now there is also another parameter called
"keep_cache_private". If set, then both authenticated and
unauthenticated GET requests for small files will not evict the data
from the buffer cache. The default values are 5 MiB and False,
respectively, so the default behavior is the same.
Bonus: the "mb_per_sync" parameter is now documented in the deployment
guide.
Change-Id: I9a11dbe861f4c23535c6aa82a9111a6fe2db2a59

also fixed bug where in error_log where identifier wasn't being set to anything meaningful, set it to the right thing
bug 612722
Change-Id: I53f237ea0db2a5b2b8979a7b41189faf1275e861

bug 786197
Change-Id: Idd73986ca4550b20dff340df4acd12f3bfc01075

metdata -> metadata
requets -> requests
Change-Id: Idc3c67e6415532e64e1d04c767f64e1d50269f44

- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.
Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0

Fixes bug 987388
Change-Id: I6eb5c45fe1f5844ad853a4ff9bc8fd23cc9abd5d

Fixes LP Bug #1008667.
Change-Id: I1e767a804b617eff8a9700c3d98b2360c040933a

bug 644075
Change-Id: I75c73a42ddd8654a39a2fd82320941199bee4363

This fixes 500 server crashes caused by requests such as:
curl -X__init__ "http://your-swift-object-server:6000/sda1/p/a/c/o"
Fixes bug 1005903
Change-Id: I6c0ad39a29e07ce5f46b0fdbd11a53a9a1010a04

Before, a really lagged cluster might not get its final report for a
deleted container database sent to its corresponding account
database. In such a case, the container database file would be
permanently deleted while still leaving the container listed in the
account database, never to be updated since the actual container
database file was gone. The only way to fix such the situation before
was to recreate and redelete the container.
Now, the container database file will not be permanently deleted
until it has sent its final report successfully to its corresponding
account database.
Change-Id: I1f42202455e7ecb0533b84ce7f45fcc7b98aeaa3

When an object server was handling concurrent GET or POST requests
from very fast clients, it would starve other connected clients. The
greenthreads responsible for servicing the fast clients would hog the
processor and only rarely yield to another greenthread.
The reason this happens for GET requests is found in
eventlet.greenio.GreenSocket, in the send() method. When you call
.send(data) on a GreenSocket, it immediately calls .send(data) on its
underlying real socket (socket._socketobject). If the real socket
accepts all the data, then GreenSocket.send() returns without yielding
to another greenthread. Only if the real socket failed to accept all
the data (either .send(data) < len(data) or by raising EWOULDBLOCK)
does the GreenSocket yield control.
Under most workloads, this isn't a problem. The TCP connection to
client X can only consume data so quickly, and therefore the
greenthread serving client X will frequently encounter a full socket
buffer and yield control, so no clients starve. However, when there's
a lot of contention for a single object from a large number of fast
clients (e.g. on a LAN connected w/10Gb Ethernet), then one winds up
in a situation where reading from the disk is slower than writing to
the network, and so full socket buffers become rare, and therefore so
do context switches. The end result is that many clients time out
waiting for data.
The situation for PUT requests is analogous; GreenSocket.recv() seldom
encounters EWOULDBLOCK, so greenthreads seldom yield.
This patch calls eventlet.sleep() to yield control after each chunk,
preventing any one greenthread's IO from blocking the hub for very
long.
This code has the flaw that it will greenthread-switch twice when a
send() or recv() does block, but since there isn't a way to find out
if a switch occurred or not, there's no way to avoid it. Since
greenlet switches are quite fast (faster than system calls, which the
object server does a lot of), this shouldn't have a significant
performance impact.
Change-Id: I8549adfb4a198739b80979236c27b76df607eebf