openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Proxy-server now requires Content-Length in the response header
when getting object and does not support chunked transferring with
"Transfer-Encoding: chunked"
This doesn't matter in normal swift, but prohibits us from putting
any middelwares to execute something like streaming processing of
objects, which can't calculate the length of their response body
before they start to send their response.
Change-Id: I60fc6c86338d734e39b7e5f1e48a2647995045ef

When reaper try to find the correct container_shard number,
but it doesn't compare local device's name, so it always
return first entry's index of local devices from the ring
to container_shard.
That causes some containers will be skip and objects under
these containers won't be delete.
Change-Id: I0e2bbdd99add86ee46e856920c0740c4aa13c77d
Closes-Bug: #1549615 

Before this commit, we performed a full read-modify-write cycle on a
partition's hashes.pkl to invalidate the relevant suffix whenever an
object PUT or DELETE completes.
Now we append invalid suffixes to a new file, "hashes.invalid",
alongside hashes.pkl. When we actually get a REPLICATE request and
need to compute the hashes, *then* we perform a full read-modify-write
on hashes.pkl and clear out hashes.invalid.
Change-Id: Ia7add438e25688b4b286f0110a4e43490e11ad75

This patch replaces some configration parameters about authtoken
middleware in proxy-server.conf, which will be deprecated.
The following parameters are to be deprecated.
 auth_admin_prefix, auth_host, auth_port, auth_protocol,
 identity_uri, admin_token, admin_user, admin_password,
 admin_tenant_name
Change-Id: I373734933189c87c4094203b0752dd3762689034

There's no RST file that uses ".. math" and thus
the pngmath Sphinx extension is not used and can
get removed.
Change-Id: I3ea4b529025c8ac3c4092f3720124647861e6668

This patch provides one container quota functional test.
Change-Id: I16354c15fc434d70f9bb6b9b1e18b09357195c83

Some of tests in test/unit/cli/test_ringbuilder doesn't assert
the exit code and unfortunately some of these passed even if the
statement fails for the assertion actually.
This patch enables to assert the exit code from ringbuider and
fixes some code/test bugs I noticed.
Change-Id: I18fa675ba8a90678e2b5ccb5f90eafab01d22787

This patch provides functional tests for X-Delete-After and X-Delete-At.
Change-Id: I55b0d0584428413aa7a31f66defec55f9b1bbcd9

This patch adds more unit tests to diminish missing pieces
of the coverage in the account_reaper unit test.
Change-Id: Ib9e875ddd1334a4a67037dcfbd42d3b008ccd4e7

This patch makes test_ringbuilder create a temporal directory,
run ring builder commands under it, and delete it after testing
for each test cases, to fix temp file leaking.
Change-Id: I6f59fe095ea6485af0e60b5a8e8fc3892e0a0f90

In proxy/controllers/base.py : ResumingGetter._get_source_and_node
self.used_source_etag should be etag of target source
instead of possible_source.
Change-Id: Id6c33e79f65cb4ad9e3249957dd57dc2da8cad98
Closes-bug: #1550734 

This renames test_object_delete_at_aysnc to
test_object_delete_at_async to match with the function
async_update which it tests.
Change-Id: I726afcbf08a3449c7af2834e573b97be378a86cd
Closes-Bug: 1550067

Change-Id: I187fb8fc8d10f950bbebc586999eced0bc117432

setup_package and teardown_package get called twice when
running functional tests using nose. With in-process mode
the first call to teardown_package clears global config
which causes the test swiftclient to raise a SkipTest error
during the second call to teardown_package.
To reproduce:
SWIFT_TEST_IN_PROCESS=1 nosetests ./test/functional/test_account.py
This patch simply tests for config existence before attempting
to create a test swiftclient in the teardown_package function.
Also fix a related issue whereby in_process flag would be reset to
False during second call to setup_package, thus causing some of the
in process cleanup in teardown_package to never be executed.
Change-Id: I074dcd3d39aa46b262632024b047556ca471e8b8

The change in 684c4c04 to improve py3 support breaks older middleware
that consumes the SwiftLogger.
Good for Swift code to not to use warn because it's deprecated.
Bad for LogAdapter to not faithfully provide the current python logging
module API (deprecated or otherwise).
Change-Id: Ib0baf66b7f33f91562867076ff0526a287d8e5a6

In common/test_utils.py, TestStatsdLogging had the majority of its
test cases calling the real socket.getaddrinfo(), which uses real
DNS. This is very slightly slower than using a mock getaddrinfo() when
the machine running the tests has functioning DNS, but on a machine
with no network connection at all, the tests are excruciatingly slow
due to timeouts.
This commit mocks things out as appropriate. There's still one user of
the real getaddrinfo(), but it's for ::1, so that's just local
resolution based on /etc/hosts.
Timing numbers for "./.unittests test.unit.common.test_utils:TestStatsdLogging":
 * network, without this patch: 1.8s
 * no network, without this patch: 221.2s (ouch)
 * network, with this patch: 1.1s
 * no network, with this patch: 1.1s
Change-Id: I1a2d6f24fc9bb928894fb1fd8383516250e29e0c

Change-Id: I247d2867c55d6bbf2288dfb74271f57c6f88b243

Change Ifcc4392ffa4df485ce3c62a35d0fcc8c93de74aa introduced
TestContainerController4Replicas as a subclass of
TestContainerController. And there is duplicated code
block to override setUp method.
This patch removed the duplicated code block.
Change-Id: Iad0b64cba52554b89dbd959475a2ce7c0d9a20eb

Currently, the ECObjectController removes the 'content-length' header.
This part is ok, except that value is being used to set
'X-Backend-Obj-Content-Length', so it is always 0. This leads to not
calling fallocate (details on bug) on a PUT since the size is 0.
This change makes use of some numbers returned from the EC Driver
get_segment_info method in order to calculate the expected on-disk
size that should be allocated. The EC controller will now set the
'X-Backend-Obj-Content-Length' value appropriately.
Co-Authored-By: Kota Tsuyuzaki
Co-Authored-By: John Dickinson
Co-Authored-By: Tim Burke
Change-Id: Ifd16c1438539e6fd9bb2dbcd053d11bea2e09fee
Fixes: bug 1532008

There are some unit test cases which assumes 200 as the response
of PUT request, but 200 is never returned in PUT.
This patch replaces 200 in PUT by 201, and adapt unit test cases
to real case.
Change-Id: I35fc7a4fe7fcc5558a9e6c6371105793c2710f3e

This patch provides unit tests to check responses for various situation
in three/four replicas env.
Change-Id: Ifcc4392ffa4df485ce3c62a35d0fcc8c93de74aa

Previously, if a user could write to (but not read from) a container,
the behavior for object POST would vary depending on whether
object_post_as_copy was enabled (403 response) or disabled (202
response).
Now, POSTs will consistently be allowed, regardless of whether fast-POST
is enabled.
Change-Id: I1d6dcbc4f5034a322a1073850fc3b059ebb1c0fa

Change-Id: I218e08446c5ffda2db7c774ec8bb9d28ce9f63e7

Consider two replicas of the same object whose ondisk files
have diverged due to failures:
 A has t2.ts
 B has t1.data, t4.meta
(The DELETE at t2 did not make it to B. The POST at t4 was
rejected by A.)
After ssync replication the two ondisk file sets will not be
consistent:
 A has t2.ts (ssync cannot POST t4.meta to this node)
 B has t2.ts, t4.meta (ssync should not delete t4.meta,
 there may be a t3.data somewhere)
Consequenty the two nodes will report different hashes for the
object's suffix, and replication will repeat, always with the
inconsistent outcome. This scenario is reproduced by the probe
test added in this patch.
(Note that rsync replication does result in (t2.ts, t4.meta)
on both nodes.)
The solution is to change the way that suffix hashes are
calculated. Currently the names of *all* files found in each
object dir are added to the hash. With this patch the
timestamps of only those files that could be used to
construct a valid diskfile are added to the hash. File
extensions are appended to the timestamp so that in most
'normal' situations the result of the hashing is the same
as before this patch. That avoids a storm of hash mismatches
when this patch is deployed in an existing cluster.
In the problem case described above, t4.meta is no longer
added to the hash, since it is not useful for constructing
a diskfile. (Note that t4.meta is not deleted because it
may become useful should a t3.data be replicated in future).
Closes-Bug: 1534276
Change-Id: I99e88b8d5f5d9bc22b42112a99634ba942415e05

Swift object-server will return 409 Conflict when the incomming
request X-Timestamp is older than the local disk file to prevent
wasted data transfer and disk space consumption. However, IFF,
the local disk file is a tombstone, current object-server will
write the data in the device and cleanup it according to timestamp
comparison result in hash_cleanup_list_dir, and then return 201
Created. That's wasted and far from the semantics in the normal case.
This patch fixes object-server to catch the DiskFileDeleted exception
at PUT request, and then, set the timestamp from tombstone as
original_timestamp to be able to compare with request timestamp.
Co-Authored-By: Kazuhiro Miyahara <miyahara.kazuhiro@lab.ntt.co.jp>
Change-Id: I078c9cb90707a3c320708e76ea42cbfa73e1ea4b
Closes-Bug: #1546865 

Reduced source code by extracting duplicated code
(swift/common/direct_client.py)
http://openqa.sed.hu/dashboard/index/5?did=1
Change-Id: I47fa6ae12c646e8e42fb0a46a017971f05d883fc
Closes-Bug: 1536067

Some doc files and swift.conf-sample were added execute permissions in past changes.
This patch remove execute permissions from them.
Change-Id: Id8844989a8321578e9207566ebd6660f5b9523f0