openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Allow Swift daemons and servers to optionally accept a directory as the
configuration parameter. Directory based configuration leverages
ConfigParser's native multi-file support. Files ending in '.conf' in the
given directory are parsed in lexicographical order. Filenames starting with
'.' are ignored. A mixture of file and directory configuration paths is not
supported - if the configuration path is a file behavior is unchanged.
 * update swift-init to search for conf.d paths when building servers
 (e.g. /etc/swift/proxy-server.conf.d/)
 * new script swift-config can be used to inspect the cumulative configuration
 * pull a little bit of code out of run_wsgi and test separately
 * fix example config bug for the proxy servers client_disconnect option
 * added section on directory based configuration to deployment guide
DocImpact
Implements: blueprint confd
Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b

Including the time inside the trans_id can be very useful for knowing
which logs to scan. I made this so the trans_id will still be the
same length (the randomness of the remaining uuid4 should be enough
for this use). I also added a convenience function for retreiving the
time information from a trans_id.
If you're wondering why I just didn't use uuid1 that embeds the time,
it's because it also embeds uuid.getnode() which "The first time this
runs, it may launch a separate program, which could be quite slow."
We could supply our own getnode value, but then we have to guarantee
its uniqueness, yada yada yada.
Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396

Increase testing coverage and make some of the test more accurate
(Some of the tests included too many responses from the account
servers that were not used by the test)
Change-Id: Ide2e0fcb89d5905e70d3111f7ac57b1cff23a99c
Fixes: Bug #1172152 

If you manage to break a proxy server so badly that it can't even
return headers, then in test/unit/proxy/test_server.py,
readuntil2crlfs() will spin forever, killing your laptop battery and
providing zero help figuring out what's wrong.
This test-only change makes it so that, should readuntil2crlfs() run
out of data before getting its two CRLFs, it raises an exception that
tells you what it did manage to get, and does so in a finite amount of
time.
Change-Id: Ieacd18ce7f4d35a4960100d5fc3a0f910cb406ac

Change-Id: I6379db52a75ed40080dd220f7fa72bcaffd0953c

Current codes will copy metadata headers when x-fresh-metadata:false, we
still need copy "x-delete-at" header and ensure expiring work at the same
time.
Change-Id: Ie31326b5f7b565e51e5aa249279bc1786f7bc847
Fixes: bug #1067528 

Towards defining the DiskFile class, or something like it, as an API
for the low level disk acesses, we push the fallocate() system call
down into the DiskFile.mkstemp() method. This allows another
implementation of DiskFile to decide to use or not use fallocate().
Change-Id: Ib4d2ee1f971e4e20e53ca4b41892c5e44ecc88d5
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Allows client-side technologies such as Flash, Java and Silverlight running
on web pages served elsewhere to interact with the Swift API.
Bug #1159960
Change-Id: I7d0533a0aaf189ac452abbd983469acb064fdca4

Previous logging made a mess of ipv6 addresses. This just makes the
colon quote-safe so it passes through unscathed. The logs will still
be backward compatible because unquote don't care:
 $ python
 >>> from urllib import quote, unquote
 >>> quote('2001:db8:85a3:8d3:1319:8a2e:370:7348')
 '2001%3Adb8%3A85a3%3A8d3%3A1319%3A8a2e%3A370%3A7348'
 >>> unquote(quote('2001:db8:85a3:8d3:1319:8a2e:370:7348'))
 '2001:db8:85a3:8d3:1319:8a2e:370:7348'
 >>> quote('2001:db8:85a3:8d3:1319:8a2e:370:7348', '/:')
 '2001:db8:85a3:8d3:1319:8a2e:370:7348'
 >>> unquote(quote('2001:db8:85a3:8d3:1319:8a2e:370:7348', '/:'))
 '2001:db8:85a3:8d3:1319:8a2e:370:7348'
Change-Id: Ia13a9bc8a320cde5c56034a7f0b645913428bf21

I just noticed tonight when adding a bunch of stuff to Swiftly that
the Bulk Delete middleware uses an account-level DELETE request,
albeit with a query parameter of bulk-delete. But, one typo and,
assuming the cluster supports it and you have access, whoops, you
just marked the account for deletion!
I put a bit of extra safety on the account deletion by requiring it
to have an empty query string.
Change-Id: Ib5df11193b04eff69d14185bd9d0607169131e7f

Change-Id: I5c615c6f32967510f09b783b1ba7089119f1d8bd

Change-Id: Ib500f5fc2e890c3d23641350815a910168bcd543

If mount_check is true (ie an SAIO with "real "devices, not loopback),
then the servers will correctly return 507 when given a nonsense path.
The first element is treated as a drive path, and that path isn't
mounted. This patch adds 507 as a valid status response to the server
check.
Change-Id: I1d1bb0ab78fd9ea17323635da7e686182fbdbf13

account_quota middleware fails to check if account_info exists
after retrieving it.
Change-Id: I1aa492d3bd713a6bf27e18ef4ec2e29f4cfb694d
Fixes: Bug #1167389 

The trans_id_suffix (default is empty) would be appended to the swift transaction
id allowing one to easily figure out from which cluster that X-Trans-Id
belongs to. This is very useful when one is managing more than one swift
cluster. Also updated sample and manpage to reflect the changes.
Change-Id: Icdf63643e9c1bde36a9ef5e3f41ee9fb20e55f5d

This patch identifies md5 collisions on objects and sends a 403
from the object server.
Credits for originating this fix are to Michael Factor.
Change-Id: I4f1b32183e2be6bbea56eaff86b9a4c7f440804a
Fix: Bug #1157454 

Extensive refactor here to consolidate what nodes are contacted for
any request. This consolidation means reads will contact the same set
of nodes that writes would, giving a very good chance that
read-your-write behavior will succeed. This also means that writes
will not necessarily try all nodes in the cluster as it would
previously, which really wasn't desirable anyway. (If you really want
that, you can set request_node_count to a really big number, but
understand that also means reads will contact every node looking for
something that might not exist.)
* Added a request_node_count proxy-server conf value that allows
 control of how many nodes are contacted for a normal request.
In proxy.controllers.base.Controller:
* Got rid of error_increment since it was only used in one spot by
 another method and just served to confuse.
* Made error_occurred also log the device name.
* Made error_limit require an error message and also documented a bit
 better.
* Changed iter_nodes to just take a ring and a partition and yield
 all the nodes itself so it could control the number of nodes used
 in a given request. Also happens to consolidate where sort_nodes is
 called.
* Updated account_info and container_info to use all nodes from
 iter_nodes and to call error_occurred appropriately.
* Updated GETorHEAD_base to not track attempts on its own and just
 stop when iter_nodes tells it to stop. Also, it doesn't take the
 nodes to contact anymore; instead it takes the ring and gets the
 nodes from iter_nodes itself.
Elsewhere:
* Ring now has a get_part method.
* Made changes to reflect all of the above.
Change-Id: I37f76c99286b6456311abf25167cd0485bfcafac

In trying to override the content-type for the convenience
call I broke COPYs and POSTs for SLO. yaaa...
Change-Id: Ifbcda6d2dd0ee43e43d62e58a90301e7afd05e27

- We allow all headers requested in preflight request. The CORS
 specification does leave the door open for this, as mentioned in
 http://www.w3.org/TR/cors/#resource-preflight-requests
 Note: Since the list of headers can be unbounded
 simply returning headers can be enough.
- This is a followup to review:
 https://review.openstack.org/#/c/24415/.
- Fixes bug 1155034.
Change-Id: If7b8f2f3a581c5209892d1ccc9f06ddb8fac92dd

Folks have actually been asking for this. I think they're sending a
DELETE TempURL to someone way ahead of time and the someone issues it
when they're ready. Honestly, I'm not entirely sure of the use case,
but having the set of methods configurable wouldn't hurt.
Change-Id: Ibdb48f8a72077b045eeedddfae4c0a1f56098d7a

Change-Id: I9002fa193a51f40523e7936e3117a2f3f2b2f7f8

Change-Id: I657c9f7a7555d90ccfa7e6ff0f5f3fcbd7eada1d

DLO is supposed to sleep 1 second per segment after 10 (default) initial
segments are loaded to prevent abuse of the system. Currently, if the user
did a range request to the 11th segment the 1 sec delay would start immediately
on the 11th segment because the skipped segments were counted as initial
segments. This path fixes that.
Change-Id: I53bbc24af54921ddc04d9cb3b704366b51b3f740

Change-Id: Ifeab56a964630bcf941e932fcbe39e6572e62975

Change-Id: Id510cceb88f062f921450a5d25d7c97b7ca5943b

A new configuration parameter is added to /etc/swift/swift.conf
[swift-hash]
swift_hash_path_prefix = 'random unique string'
New installations are advised to set this parameter to a random secret,
which would not be disclosed ouside the organization.
The same secret needs to be used by all swift servers of the same cluster.
Existing installations should set this parameter to an empty string
(the default)
DocImpact
Fixes: Bug #1157454
Change-Id: I63b10d0b7d6dd3f74e0f10bb41b5f240fa03578a

Use for reading metadata in unit tests function from object-server.
Change-Id: I2bfeb76fdd775442a0e614fef740b0987fba4a22
Fixes: bug #1079131 

Added 411 responses in bulk and slo when needed.
Make X-Static-Large-Object an illegal header with slo installed- somehow
that got lost with some refactor I did.
Change-Id: I986c1656658f874172860469624118cc63bff9bc

If X-Copy-From is a manifest object, it can be bigger than MAX_FILE_SIZE.
Currently this fails with 503. It should be 413.
Bug #1158279
Change-Id: I7ec16088046c84e42d5be9c865e1338eb07845b6