openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: If537981e8deadd9c3528dcb30a15011c7781e334

Change-Id: I798d08722c90327c66759aa0bb4526851ba38d41

The ring has some optimizations in get_more_nodes() so that it can
find handoffs that span all the regions/zones/et cetera and then stop
looking. The stopping is the important part.
Previously, it would quickly find a handoff in each unused region,
then spend way too long looking for more unused regions; the same was
true for zones, IPs, and so on. Thus, in commit 9cd7c6c, we started
counting regions and zones, then stopping when we found them all.
This count included all regions and zones in the ring, regardless of
whether or not there were actually any parts assigned or not. In rings
with an empty region, i.e. a region for which there are only
zero-weight devices, get_more_nodes() would be very slow.
This commit ignores devices with no assigned partitions when counting
regions, zones, and so forth, thus greatly speeding things up.
The output of get_more_nodes() is unchanged. This is purely an
optimization.
Closes-Bug: 1534303
Change-Id: I4a5c57205e87e1205d40fd5d9458d4114e524332

Change-Id: Id6329c863dacb189fccfc304453ed7b6f9607c14

Change-Id: Ic451c5e0b686509f8982ed1bf65a223a2d77b9a0

It's not consistent now for example local time in replication part and
UTC time at begging of line. Use _ptime in swift-recon for all time
printing and this function returns UTC now.
Change-Id: I732d9851db157130a08e825e8093b7e244b63e9c

Previously, the object-auditor would always use a (replication)
DiskFileManager when walking through AuditLocations, which would cause
it to skip EC fragment archives with a warning like:
 Unexpected file <hash_path>/1452557187.03610#3.data:
 Invalid Timestamp value in filename '1452557187.03610#3.data'
Now, the AuditLocation's policy will be used to find an appropriate
manager to get the diskfile. Additionally, several .commit()s were added
to the auditor tests so the .durable will be written, which is required
when auditing EC fragment archives.
Change-Id: I960e7d696fd9ad704ca1872b4ac821f9078838c7
Closes-Bug: 1533002

The patch moves the MemcacheConnPool._get_addr() method a function in
swift.common.utils. The function is renamed to parse_socket_string()
and the documentation is updated accordingly. The test for it has also
been moved.
Change-Id: Ida65b2fded28d0a059e668646f5b89714298f348

In cases when the memcache address is invalid, we should fail early.
This patch addresses the cases when the IPv6 addresses are not
enclosed in "[]". It does not, however, fix the case of an invalid
hostname. These improvements could also be added to the _get_addr()
method.
Change-Id: I4743dcda45a1fc1640989325c4a2e1fea591fc69

Change-Id: I20d91c1e276014eaf210fa9eb43788bc17f4e8df

Change-Id: If673afa2b61a3e388612debf4860d561960963a3

Multiple etags can be provided on an if-match or if-none-match
request. This is currently being tested in the unit tests, but not
in the functional tests. Since these etags can be modified by
middleware, we need functional tests to assert multiple-etag
requests are handled correctly.
Change-Id: Idc409c85e8aa82b59dc2bc28af6ca2617de82699

Change-Id: I05af7d837af00cd1059a8912df05c131a52ffb90

Previously, attempting to PUT a new object with an X-Timestamp header
less than or equal to zero (ie, for a timestamp on or before 1970年01月01日
00:00:00) would cause the object-server to 500.
While we're at it, cap X-Timestamp at 9999999999 (2286年11月20日 17:46:40)
so we don't get an eleventh digit before the decimal point.
Closes-Bug: 1532471
Change-Id: I23666ec8a067d829eaf9bfe54bd086c320b3429e

swift-ring-builder currently only displays min_part_hours and
not the amount of time remaining before a rebalance can occur.
This information is readily available and has been displayed
as a quality of life improvement.
Additionally, a bug where the time since the last rebalance
was always updated when rebalance was called regardless of
if any partitions were reassigned. This can lead to partitions
being unable to be reassigned as they never age according to
the time since last rebalance.
Change-Id: Ie0e2b5e25140cbac7465f31a26a4998beb3892e9
Closes-Bug: #1526017 

The patch removes the assumption of IPv4-only addresses in the
MemcacheConnPool. The changes are around address handling.
Namely, if a server is specified with an address
[<address>]:port (port is optional), it is assumed to be an IPv6
address [1]. If an IPv6 address is specified without "[]", an exception
is raised as it is impossible to parse such addresses correctly.
For testing, memcache can be configured to listen on the link-local,
unique-local, or ::1 (equivalent to 127.0.0.1) addresses. Link-local
addresses are assigned by default to each interface and are of the form
"fe80::dead:beef". These addresses require a scope ID, which would look
like "fe80::dead:beef%eth0" (replacing eth0 with the correct interface).
Unique-local addresses are any addresses in the fc00::/7 subnet. To add
a ULA to an interface use the "ip" utility. For example:
"ip -6 address add fc01::dead:beef dev eth0". Lastly, and probably
simplest, memcache can be configured to listen on "::1". The same
address would be used in the swift configuration, e.g. "[::1]:11211".
Note: only memcached version 1.4.25 or greater supports binding to an
IPv6 address.
Fixes #1526570
[1] IPv6 host literals:
https://tools.ietf.org/html/rfc3986#section-3.2.2
Change-Id: I8408143c1d47d24e70df56a08167c529825276a2

Change-Id: I1980ebdd9dc89848173d8ca2fe2afb74029dcfa2
Closes-Bug: 1532276

Swift now supports Range header for COPY (or PUT with X-Copy-From)
to make a partial copied object. This patch adds the note to show
it obviously supported in Swift community.
Change-Id: I6bf28f0932c90e7b305cd61aabce4ed028ae691e
Partial-Bug: #1532126 

swift/container/sync.py uses swift.common.internal_client.delete_object
and put_object and expected these methods raise ClientException.
But delete_object and put_object never raise the exception so this patch
raises ClientException when urllib2 library raises HTTPError.
Co-Authored-By: Eran Rom <eranr@il.ibm.com>
Closes-Bug: #1419901
Change-Id: I58cbf77988979a07998a46d9d81be84d29b0d9bf

When processing keys where the names start with the delimiter
character, swift should list only the delimiter character. To get the
list of nested keys, the caller should also supply the prefix which is
equal to the delimiter.
Added a functional test and unit tests to verify this behavior.
Fixes Bug: 1475018
Change-Id: I27701a31bfa22842c272b7781738e8c546b82cbc

os.popen() is deprecated since version 2.6. Resolved with use of
subprocess module.
Change-Id: I4409cdd9edbc1a26d6f99c125c9100fadda5d758
Partial-Bug: #1529836 

This change introduces a sync_store which holds only containers that
are enabled for sync. The store is implemented using a directory
structure that resembles that of the containers directory, but has
entries only for containers enabled for sync.
The store is maintained in two ways:
1. Preemptively by the container server when processing
PUT/POST/DELETE operations targeted at containers with
x-container-sync-key / x-container-sync-to
2. In the background using the containers replicator
whenever it processes a container set up for sync
The change updates [1]
[1] http://docs.openstack.org/developer/swift/overview_container_sync.html
Change-Id: I9ae4d4c7ff6336611df4122b7c753cc4fa46c0ff
Closes-Bug: #1476623 

Change-Id: Iad410a2be4f9a2cd5c53e860b9f91993aa7f2369
Closes-Bug: #1531173 

Change-Id: I8479c85cb8821c48b5da197cac37c80e5c1c7f05

This option send SIGKILL to daemon after kill_wait period.
When daemon hangs and doesn't respond to SIGTERM/SIGHUP
there is no way to stop it using swift-init now. Classic
init scripts in Linux kills hanged process after grace
period and this patch add same behaviour. This is most
usefull when using "restart" on hanged daemon.
Change-Id: I8c932b673a0f51e52132df87ea2f4396f4bba9d8

LogAdapter object has no attribute 'warn' but has attribute
'warning'.
Closes-Bug: #1529321
Change-Id: I0e0bd0a3dbc4bb5c1f0b343a8809e53491a1da5f

account-replicator and container-replicator outputted the
following info in log.
info account-replicator: 17 successes, 4 failures
info container-replicator: 178 successes, 29 failures
But there is no output info for object-replicator. This info
is important to check replicator's status from log.
So this patch adds object-replicator's succeeded and failed
counts in log like account-replicator and container-replicator.
Change-Id: I7fbaf161540bdb06c0a65a1032aae83764bd970c

DeprecationWarning: The 'warn' method is deprecated, use 'warning'
instead
Change-Id: I35df44374c4521b1f06be7a96c0b873e8c3674d8

There are following problems with Access-Control-Expose-Headers.
* If headers in X-Container-Meta-Access-Control-Expose-Headers are
 configured, the headers are kept with case-sensitive string.
 Then a CORS request comes, the headers are merged into
 Access-Control-Expose-Headers as case-sensitive string even if
 there is a same header which is not case-sensitive string.
* Access-Control-Expose-Headers is handled by a list.
 If X-Container/Object-Meta-XXX is configured in container/object
 and X-Container-Meta-Access-Control-Expose-Headers, same header
 is listed in Access-Control-Expose-Headers.
This patch provides a fix for the problems.
Change-Id: Ifc1c14eb3833ec6a851631cfc23008648463bd81