openstack/swift - swift - OpenDev: Free Software Needs Free Tools

The crossdomain doc was named *.xml instead of *.rst causing it to not
get built or included in the toctree where it was supposed to.
The apache deployment guide wasn't linked to from anywhere, so I added
it under the normal deployment guide.
Change-Id: I817a1f2ca1ed7913e8ea5155cc1fac07caf0b637

When swift-bench is run in direct mode, don't try to delete the
containers which weren't created.
Fixes bug 1177960.
Change-Id: Ice07e8729bb776e2b215894cf95fb80b64167a8d

Change-Id: I8ea0ff86518d453597faae44ec3918298e2d5147

This allows users to rotate their TempURL keys without invalidating
all existing signed URLs. This is handy if you have multiple systems
generating signed URLs, but you want to change your keys for some
reason (e.g. keys compromised, company policy, general paranoia).
Both the first and second keys are optional, so existing accounts'
signed URLs will continue to work as before.
This commit does change the memcache key used to store the fetched
TempURL keys. This is because we were storing the old key as a string
in memcached, but the new one is a list of keys. Since the key cache
lifetime here is only 60 seconds, it doesn't seem like too big a deal
to completely flush the TempURL cache.
Also, this commit adds caching of a negative TempURL result. If the
account HEAD reveals no TempURL keys at all, that result is now stored
for 60 seconds the same way that a positive result would be.
Change-Id: I40a02bd607283fbce11aa52a9bb8a5846ab17f5e

Change-Id: I431cac1e9818a335ff8d20288ab6acf39d6b6d5e

Currently if you have a container with objects named
x/y and x0 and you use delimiter=/ in your query, x0
will be excluded from the results. This patch resolves
this problem.
To replicate the original issue, create a container
called 'test' with the following objects:
x/y
x0
x1
Then do a GET with 'accnt/test?delimiter=/' and the
following is returned:
x/
x1
i.e. x0 is missing from output.
Change-Id: If196e3075612b121ef8da4a9128167d00a248c27

dnspython was added recently to the list of requirments.
This patch completes the change:
1. The new requirement is documented
2. The need for tox --recreate is documented
(assuming there is no way to automate 2 above)
Change-Id: I49f0cb4b01b79dd7cf4329984a2f8416378ebe19
Fixes: Bug #1176627 

It already skips IPv4 addresses, and since IPv6 is the future of the
Internet*, we should probably do the right thing with those too.
* IPv6: Just two years away for over fifteen years!
Change-Id: I54f1db4e936fd38d05ac8b5c709efba76525b9d2

cname_lookup cannot resolve IP addresses as CNAME records, and therefore
should not attempt to resolve the host in that case. The middleware is
skipped when the host is an IP address.
Change-Id: I6961ec205e771116ace1ebcb8c088f3116eb38f0
Fixes: bug #1172289 

Broke this out into a separate commit since adding five characters
caused a few lines to get re-wrapped, adding more diffs.
See John's comment from Patch Set #3 for the motivation for this
change in review https://review.openstack.org/27149.
Change-Id: I1edf2cb468dcda0b781569161ada6e4016c1141c
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Before when iterating using this object, the call to the underlying file object
was made without a chunk_size. This is bad because this would cause the entire
contents of the file to be read.
Change-Id: I9956e5d2d693a6260252fff331d4f78f70179a6c

Change-Id: I3d4a31111c3044da06611405ce80f208ef8a0ce3

Forgot to update dicts with changeset I4a89afad, which updated lists.
Change-Id: Ieca71b9c90ee5dae83a43f6851b6b8b2924bcb8e
Fixes: bug 1174809

Addressing bug 1174809, changes use of mutable lists as as default
arguments and defaults them within the function. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.
To my knowledge there aren't known cases of the current use causing
specific issues, but even stylistically needs addressing to avoid
problems in the future. I couldn't find any comments or related
historical commit messages indicating the current use is meant to take
advantage of this behavior in Python.
Change-Id: I4a89afada08b2ce220724f585631a9e2072bf1bd
Fixes: bug 1174809

Allow Swift daemons and servers to optionally accept a directory as the
configuration parameter. Directory based configuration leverages
ConfigParser's native multi-file support. Files ending in '.conf' in the
given directory are parsed in lexicographical order. Filenames starting with
'.' are ignored. A mixture of file and directory configuration paths is not
supported - if the configuration path is a file behavior is unchanged.
 * update swift-init to search for conf.d paths when building servers
 (e.g. /etc/swift/proxy-server.conf.d/)
 * new script swift-config can be used to inspect the cumulative configuration
 * pull a little bit of code out of run_wsgi and test separately
 * fix example config bug for the proxy servers client_disconnect option
 * added section on directory based configuration to deployment guide
DocImpact
Implements: blueprint confd
Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b

Including the time inside the trans_id can be very useful for knowing
which logs to scan. I made this so the trans_id will still be the
same length (the randomness of the remaining uuid4 should be enough
for this use). I also added a convenience function for retreiving the
time information from a trans_id.
If you're wondering why I just didn't use uuid1 that embeds the time,
it's because it also embeds uuid.getnode() which "The first time this
runs, it may launch a separate program, which could be quite slow."
We could supply our own getnode value, but then we have to guarantee
its uniqueness, yada yada yada.
Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396

The new max_clients parameter allows one full control over the maximum
number of client requests that will be handled by a given worker for
any of the proxy, account, container or object servers.
Lowering the number of clients handled per worker, and raising the
number of workers can lessen the impact that a CPU intensive, or
blocking, request can have on other requests served by the same
worker.
If the maximum number of clients is set to one, then a given worker
will not perform another accept(2) call while processing, allowing
other workers a chance to process it.
DocImpact
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Change-Id: Ic01430f7a6c5ff48d7aa349dc86a5f8ac463a420

If we set device's weight to zero, currently balance will be set
special value(999.99) until zero weighted device return all
its partitions. So we cannot check balance has changed.
Thus we need to check balance or last_balance is special value.
Change-Id: I5b7db8b8e48db0c4771c51a764bda689869817d5
Fixes: bug #1171731 

Increase testing coverage and make some of the test more accurate
(Some of the tests included too many responses from the account
servers that were not used by the test)
Change-Id: Ide2e0fcb89d5905e70d3111f7ac57b1cff23a99c
Fixes: Bug #1172152 

If you manage to break a proxy server so badly that it can't even
return headers, then in test/unit/proxy/test_server.py,
readuntil2crlfs() will spin forever, killing your laptop battery and
providing zero help figuring out what's wrong.
This test-only change makes it so that, should readuntil2crlfs() run
out of data before getting its two CRLFs, it raises an exception that
tells you what it did manage to get, and does so in a finite amount of
time.
Change-Id: Ieacd18ce7f4d35a4960100d5fc3a0f910cb406ac

Change-Id: I6379db52a75ed40080dd220f7fa72bcaffd0953c

Current codes will copy metadata headers when x-fresh-metadata:false, we
still need copy "x-delete-at" header and ensure expiring work at the same
time.
Change-Id: Ie31326b5f7b565e51e5aa249279bc1786f7bc847
Fixes: bug #1067528 

Towards defining the DiskFile class, or something like it, as an API
for the low level disk acesses, we push the fallocate() system call
down into the DiskFile.mkstemp() method. This allows another
implementation of DiskFile to decide to use or not use fallocate().
Change-Id: Ib4d2ee1f971e4e20e53ca4b41892c5e44ecc88d5
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Allows client-side technologies such as Flash, Java and Silverlight running
on web pages served elsewhere to interact with the Swift API.
Bug #1159960
Change-Id: I7d0533a0aaf189ac452abbd983469acb064fdca4

The lack of any documentation for the auth_version was observed
by Red Hat QA, see rhbz#955247.
Change-Id: I0afc04d848082a80480724bdbf99031e4b99e38e

swift_hash_path_prefix
Change-Id: I60f5f3d4083937a03ecb7ed531185c617ea08016

Change-Id: I3568d4dc1900e6ddb4860589ca6a7b7039cc8c2d
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Support separate replication ip address:
- Added new function in utils. This function provides ability
 to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
 Replication process uses new function now.
Replication network parameters:
- Replication network fields (replication_ip, replication_port)
 support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
 functions.
Implementation of replication servers:
- Separate replication servers use the same code as normal replication
 servers, but with replication_server parameter = True. When using a
 separate replication network, the non-replication servers set
 replication_server = False. When there is no separate replication
 network (the default case), replication_server is not included in the config.
DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network

remove duplicated key from swift-bench bin; and also add .idea to gitignore
Change-Id: I31360489716d03324b8d2bb1c51e282393069c13

Previous logging made a mess of ipv6 addresses. This just makes the
colon quote-safe so it passes through unscathed. The logs will still
be backward compatible because unquote don't care:
 $ python
 >>> from urllib import quote, unquote
 >>> quote('2001:db8:85a3:8d3:1319:8a2e:370:7348')
 '2001%3Adb8%3A85a3%3A8d3%3A1319%3A8a2e%3A370%3A7348'
 >>> unquote(quote('2001:db8:85a3:8d3:1319:8a2e:370:7348'))
 '2001:db8:85a3:8d3:1319:8a2e:370:7348'
 >>> quote('2001:db8:85a3:8d3:1319:8a2e:370:7348', '/:')
 '2001:db8:85a3:8d3:1319:8a2e:370:7348'
 >>> unquote(quote('2001:db8:85a3:8d3:1319:8a2e:370:7348', '/:'))
 '2001:db8:85a3:8d3:1319:8a2e:370:7348'
Change-Id: Ia13a9bc8a320cde5c56034a7f0b645913428bf21

I just noticed tonight when adding a bunch of stuff to Swiftly that
the Bulk Delete middleware uses an account-level DELETE request,
albeit with a query parameter of bulk-delete. But, one typo and,
assuming the cluster supports it and you have access, whoops, you
just marked the account for deletion!
I put a bit of extra safety on the account deletion by requiring it
to have an empty query string.
Change-Id: Ib5df11193b04eff69d14185bd9d0607169131e7f

Drop the cache (fadvise) after calling fsync to avoid redundant work
in the kernel visiting pages that are dirty. By dropping cache after
fsync(), the pages will be clean so can be easily removed from the
cache.
Change-Id: I3013d2ac1e27bbe43e58a81638d04805f5e2ae6e
Signed-off-by: Peter Portante <peter.portante@redhat.com>