openstack/swift - swift - OpenDev: Free Software Needs Free Tools

If not explicitly configured the versioned_writes middleware
should be auto-inserted in the pipeline after slo and dlo, which
is where the versioned_writes filter section's comments say it
should be in proxy-server.conf-sample. At the moment it can end up
being placed ahead of slo and dlo if they have been explicitly
configured, which results in the linked bug manifesting.
Closes-Bug: #1537042
Change-Id: I6ac95a331f4ef0d4887311940acc6f8bc00fb4eb

Change-Id: I315f8b554bb9e96659b455f4158f074961bd6498

Change-Id: I926d7aaa9df093418aaae54fe26e8f7bc8210645

Change-Id: Idd0ff9e70abc0773be183c37cd6125fe852da7c0

When a client disconnected while consuming an SLO or DLO GET response,
the proxy would leak a socket. This could be observed via strace as a
socket that had shutdown() called on it, but was never closed. It
could also be observed by counting entries in /proc/<pid>/fd, where
<pid> is the pid of a proxy server worker process.
This is due to a memory leak in SegmentedIterable. A SegmentedIterable
has an 'app_iter' attribute, which is a generator. That generator
references 'self' (the SegmentedIterable object). This creates a
cyclic reference: the generator refers to the SegmentedIterable, and
the SegmentedIterable refers to the generator.
Python can normally handle cyclic garbage; reference counting won't
reclaim it, but the garbage collector will. However, objects with
finalizers will stop the garbage collector from collecting them* and
the cycle of which they are part.
For most objects, "has finalizer" is synonymous with "has a __del__
method". However, a generator has a finalizer once it's started
running and before it finishes: basically, while it has stack frames
associated with it**.
When a client disconnects mid-stream, we get a memory leak. We have
our SegmentedIterable object (call it "si"), and its associated
generator. si.app_iter is the generator, and the generator closes over
si, so we have a cycle; and the generator has started but not yet
finished, so the generator needs finalization; hence, the garbage
collector won't ever clean it up.
The socket leak comes in because the generator *also* refers to the
request's WSGI environment, which contains wsgi.input, which
ultimately refers to a _socket object from the standard
library. Python's _socket objects only close their underlying file
descriptor when their reference counts fall to 0***.
This commit makes SegmentedIterable.close() call
self.app_iter.close(), thereby unwinding its generator's stack and
making it eligible for garbage collection.
* in Python < 3.4, at least. See PEP 442.
** see PyGen_NeedsFinalizing() in Objects/genobject.c and also
 has_finalizer() in Modules/gcmodule.c in Python.
*** see sock_dealloc() in Modules/socketmodule.c in Python. See
 sock_close() in the same file for the other half of the sad story.
This closes CVE-2016-0738.
Closes-Bug: 1493303
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: Ib86c4c45641485ce1034212bf6f53bb84f02f612

Change-Id: I34514525b606cf82767ddce7769bc42fa5457717

Deleted comment about parameter part power in Class FakeRing as its
behavior got dropped in I8bfc388a04eff6491038991cdfd7686c9d961545.
Change-Id: Iec7d2565a77e48493b0056021066d8d8eab65d0b
Closes-Bug: #1488704 

IPv6 support in Swift is dependent on IPv6 support in eventlet.
eventlet itself only claims support for IPv6 post v0.17
(https://github.com/eventlet/eventlet/issues/8). This update matches
the OpenStack global requirements version.
Change-Id: I9d8433cdd3bf7d7a93b8f50b991cc21721a80d22

Change-Id: I86d423309f0b2091ee2e82b2245caf925b6a75ef
Closes-Bug: #1528189 

Previously, we gave no indication of when a token would expire. Users
would have to just use it until it stopped working, then re-auth.
Now, a successful auth response will include a new header,
X-Auth-Token-Expires, with the number of seconds remaining until the
token is invalid. This allows the client to attempt to re-auth before
sending a request that will definitely fail.
For comparison, swauth already uses the X-Auth-Token-Expires header with
identical semantics. Additionally, Keystone (v2 and v3) already exposes
expiration times in its JSON responses. The security impact should be
minimal.
Change-Id: I5a4a74276bc0df6dda94e4bc150065c0d77de0eb

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure
Change-Id: Ic416c9afc8a1c76f552803a7c70fc905eda7b3cb

Change-Id: I798d08722c90327c66759aa0bb4526851ba38d41

The ring has some optimizations in get_more_nodes() so that it can
find handoffs that span all the regions/zones/et cetera and then stop
looking. The stopping is the important part.
Previously, it would quickly find a handoff in each unused region,
then spend way too long looking for more unused regions; the same was
true for zones, IPs, and so on. Thus, in commit 9cd7c6c, we started
counting regions and zones, then stopping when we found them all.
This count included all regions and zones in the ring, regardless of
whether or not there were actually any parts assigned or not. In rings
with an empty region, i.e. a region for which there are only
zero-weight devices, get_more_nodes() would be very slow.
This commit ignores devices with no assigned partitions when counting
regions, zones, and so forth, thus greatly speeding things up.
The output of get_more_nodes() is unchanged. This is purely an
optimization.
Closes-Bug: 1534303
Change-Id: I4a5c57205e87e1205d40fd5d9458d4114e524332

Change-Id: Id6329c863dacb189fccfc304453ed7b6f9607c14

Change-Id: Ic451c5e0b686509f8982ed1bf65a223a2d77b9a0

It's not consistent now for example local time in replication part and
UTC time at begging of line. Use _ptime in swift-recon for all time
printing and this function returns UTC now.
Change-Id: I732d9851db157130a08e825e8093b7e244b63e9c

Previously, the object-auditor would always use a (replication)
DiskFileManager when walking through AuditLocations, which would cause
it to skip EC fragment archives with a warning like:
 Unexpected file <hash_path>/1452557187.03610#3.data:
 Invalid Timestamp value in filename '1452557187.03610#3.data'
Now, the AuditLocation's policy will be used to find an appropriate
manager to get the diskfile. Additionally, several .commit()s were added
to the auditor tests so the .durable will be written, which is required
when auditing EC fragment archives.
Change-Id: I960e7d696fd9ad704ca1872b4ac821f9078838c7
Closes-Bug: 1533002

Change-Id: I20d91c1e276014eaf210fa9eb43788bc17f4e8df

Change-Id: If673afa2b61a3e388612debf4860d561960963a3

Multiple etags can be provided on an if-match or if-none-match
request. This is currently being tested in the unit tests, but not
in the functional tests. Since these etags can be modified by
middleware, we need functional tests to assert multiple-etag
requests are handled correctly.
Change-Id: Idc409c85e8aa82b59dc2bc28af6ca2617de82699

Previously, attempting to PUT a new object with an X-Timestamp header
less than or equal to zero (ie, for a timestamp on or before 1970年01月01日
00:00:00) would cause the object-server to 500.
While we're at it, cap X-Timestamp at 9999999999 (2286年11月20日 17:46:40)
so we don't get an eleventh digit before the decimal point.
Closes-Bug: 1532471
Change-Id: I23666ec8a067d829eaf9bfe54bd086c320b3429e