openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Object-expirer changes order of expiration tasks to avoid deleting
objects in a certain container continuously.
To make review for expirer's task queue update patch [1] easy,
this patch refactors the implementation of the order change. In this
patch, the order change is divided as a function.
In [1], there will be two implementations for legacy task queue
and for general task queue. The two implementations have similar
codes. This patch helps to avoid copying codes in the two implementations.
Other than dividing function, this patch tries to resolve:
- Separate container iteration and object iteration to avoid the generator
 termination with (container, None) tuple.
- Using Timestamp class for delete_timestamp to be consist with other modules
- Change yielded delete task object info from tuple to dict because that
 includes several complex info (e.g. task_container, task_object,
 and target_path)
- Fix minor docs and tests depends on the changes above
[1]: https://review.openstack.org/#/c/517389
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: Ibf61eb1f767a48cb457dd494e1f7c12acfd205de

This patch just fixing a typo.
Change-Id: I8a3c5e86b39e442157874a426d664fce33c9d36a

The object reconstructor has a handoffs-only mode that is very useful
when a cluster requires rapid rebalancing, like when disks are nearing
fullness. This mode's goal is to remove handoff partitions from disks
without spending effort on primary partitions. The object replicator
has a similar mode, though it varies in some details.
This commit adds a handoffs-only mode to the account and container
replicators.
Change-Id: I588b151ee65ae49d204bd6bf58555504c15edf9f
Closes-Bug: 1668399

If a cluster operator has some tooling that makes directories in
/srv/node/<disk>/accounts, then the account replicator will treat
those directories as partition dirs and may remove empty
subdirectories contained therein. This wastes time and confuses the
operator.
This commit makes DB replicators skip partition directories whose
names don't look like positive integers. This doesn't completely avoid
the problem since an operator can still use an all-digit name, but it
will skip directories like "tmp21945".
Change-Id: I8d6682915a555f537fc0ce8c39c3d52c99ff3056

Generally, we shouldn't compare integers with "is". It happens to work
because (a) CPython has only one instance of each integer betwen -5
and 256, and (b) errno.ENOTEMPTY == 66 on Linux, but it's better not
to rely on those details.
Change-Id: I494a4adbb2bb35adcf83b2d23e790c1b220e75c7

If we're calling the script with any arguments, --pretty will not
be passed to ostestr.
Also redirected cd commands' output to /dev/null in .functests.
Change-Id: I6e7e391c7e1659b86ab12eae4362b565218917b2

Change-Id: I805cf220aee4af69bf4984cc148d69520e958073

I can't imagine us *not* having a py3 proxy server at some point, and
that proxy server is going to need a ring.
While we're at it (and since they were so close anyway), port
* cli/ringbuilder.py and
* common/linkat.py
* common/daemon.py
Change-Id: Iec8d97e0ce925614a86b516c4c6ed82809d0ba9b

Otherwise, you could start running out of file handles when running all
tests. Previously, I'd regularly exceed 1k fds and then all subsequent
tests would fail; now, it maxes out around 90.
Change-Id: Ib53f15b5d2be95c70cce8903ea0ef5c334479837

Change-Id: I14dd433ed7c7fe789d5d04defbf9eec5bffc301a

Change-Id: Ic375b6c6ebf3f66860b065785e75b8d47552be28

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I1409c8eaca4fbe1170d2e998b35e7fdacf4b86b4

Change-Id: I0982b0046a16fda0a39d9b31402b2e4b3160a5c4

orphans processes sometimes cause probe test failures so
get rid of them before each test.
Change-Id: I4ba6748d30fbb28371f13aa95387c49bc8223402

Ran into an eventlet bug[0] while integration Swift/Barbican
in TripleO. It is very similar to a previous bug related
to keystonemiddleware[1]. Suggestion from urllib3[2] is to
patch eventlet "as early as possible". Traceback[3] shows that
urllib3 is being imported before the eventlet patch, so moved
the patch to before the loadapp call.
[0] - http://paste.openstack.org/show/658046/
[1] - https://bugs.launchpad.net/swift/+bug/1662473
[2] - https://github.com/shazow/urllib3/issues/1104
[3] - https://gist.github.com/thiagodasilva/12dad7dc4f940b046dd0863b6f82a78b
Change-Id: I74e580f31349bdefd187cc5d6770a7041a936bef

Change-Id: I7b2dd19f18cbfac3bc3ea763cd3333e3fb68f1cd

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: Ib339f89f634a22ea874dcf858c43a14d1145ed77

Closes-Bug: #1747689
Change-Id: Ief6bd0ba6cf675edd8ba939a36fb9d90d3f4447f

It was rare (saw it once in 10k runs running locally), but it's
ocassionally blown up in the gate [1]. With this, no fails locally even
after 100k runs.
[1] http://logs.openstack.org/11/538011/3/gate/swift-tox-py27/06c06f0/job-output.txt.gz#_2018年02月07日_03_29_09_578389
Change-Id: I7701d2db2ec82b48559c5b74a2e08c4403fd5dec
Related-Change: Ia126ad6988f387bbd2d1f5ddff0a56d457a1fc9b

Zuul no longer requires the project-name for in-repo configuration.
Omitting it makes forking or renaming projects easier.
Change-Id: I2226881d18e69bf25ad93ee8b8db67248e14f697

Change-Id: I577d169022916676a20a9ac24c7cc7b63ae46778

We had a pair of large, complicated iterators to handle fetching all
the segment data, and they were hard to read and think about. I tried
to break them out into some simpler pieces:
 * one to handle coalescing multiple requests to the same segment
 * one to handle fetching the bytes from each segment
 * one to check that the download isn't taking too long
 * one to count the bytes and make sure we sent the right number
 * one to catch errors and handle cleanup
It's more nesting, but each level now does just one thing.
Change-Id: If6f5cbd79edeff6ecb81350792449ce767919bcc

As the bug 1743310 reported,if we list the swift processes ,we will 
see a zombie process every one minute.The zombie processes numbers may 
be more than one. 
The related code as follows: swift/obj/auditor.py:386~397
 while pids:
 pid = os.wait()[0]
 # ZBF scanner must be restarted as soon as it finishes
 # unless we're in run-once mode
 if self.conf_zero_byte_fps and pid == zbf_pid and \
 len(pids) > 1 and not once:
 kwargs['device_dirs'] = override_devices
 # sleep between ZBF scanner forks
 self._sleep()
 zbf_pid = self.fork_child(zero_byte_fps=True, **kwargs)
 pids.add(zbf_pid)
 pids.discard(pid)
The pids list includes a zbf pid and one or more pids with mode 
(ALL - parallel, objectXXX).If the zbf process is the last one 
finished,then the second zbf process will not be forked.Conversely, 
the zbf process will be forked again,before the fork procedure,the main 
process will sleep for self.interval seconds,30 seconds by default,
during the self.interval time,if a non zbf process finished,this process 
will become a zombie,because the main process is in sleep.
In this solution,we move the sleep from whileloop to the subprocess,and
the main process will not be blocked in whileloop,so the subprocess
will be recovered in time.
Closes-Bug: #1743310
Change-Id: I61c766aa2a1c4bad0247a44a8e78ef38d9f3ae47

In expirer's unit tests, FakeInternalClient instances simulates
expirer's task queue behavior. But get_account_info method of
the FakeInternalClient returns container count = 1 and object
count = 2, even if it simulate different count of containers or
objects.
This patch fixes the behavior. The return values of get_account_info
will be equal to simulated container and object counts.
This patch will make review for expirer's task queue upgrade patch [1]
more easy.
[1]: https://review.openstack.org/#/c/517389
Change-Id: Id5339ea7e10e4577ff22daeb91ec90f08704c98d

The tests were carefully setting up a mock for the 'http_connect'
function in the ratelimit module, but there is no such function
imported by the ratelimit module.
As far as I can tell, this has been the case since the ratelimit
middleware first appeared in 72d40bd (Mon Oct 4 14:11:48 2010 -0700).
Change-Id: If047184c6435aa1647050f50b499dc9feff4318d

In expirer's unit tests, fake InternalClient classes are defined
and its instance simulates expirer's task queue behaviors.
To make review for expirer's task queue update patch [1] easy,
this patch refactors the implementation of the fake InternalClient
classes. In this patch, unit tests are refactored by the following
two approaches:
 #1: Summarizing duplicated fake InternalClient implementation
 #2: Make task account name variable
The #2 approach is for multiple task accounts in [1].
The patch [1] will be rebased after this patch merged.
[1]: https://review.openstack.org/#/c/517389
Change-Id: I10a7151cfdd43460ad38c47f672d3c31b77e7990

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I6e420ea5241838192571dd9d857cca4056699314

... and add support for SHA-256 and SHA-512 by default. This allows us
to start moving toward replacing SHA-1-based signatures. We've known
this would eventually be necessary for a while [1], and earlier this
year we've seen SHA-1 collisions [2].
Additionally, allow signatures to be base64-encoded, provided they start
with a digest name followed by a colon. Trailing padding is optional for
base64-encoded signatures, and both normal and "url-safe" modes are
supported. For example, all of the following SHA-1 signatures are
equivalent:
 da39a3ee5e6b4b0d3255bfef95601890afd80709
 sha1:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
 sha1:2jmj7l5rSw0yVb/vlWAYkK/YBwk
 sha1:2jmj7l5rSw0yVb_vlWAYkK_YBwk=
 sha1:2jmj7l5rSw0yVb_vlWAYkK_YBwk
(Note that "normal" base64 encodings will require that you url encode
all "+" characters as "%2B" so they aren't misinterpretted as spaces.)
This was done for two reasons:
 1. A hex-encoded SHA-512 is rather lengthy at 128 characters -- 88
 isn't *that* much better, but it's something.
 2. This will allow us to more-easily add support for different
 digests with the same bit length in the future.
Base64-encoding is required for SHA-512 signatures; hex-encoding is
supported for SHA-256 signatures so we aren't needlessly breaking from
what Rackspace is doing.
[1] https://www.schneier.com/blog/archives/2012/10/when_will_we_se.html
[2] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
Change-Id: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Related-Bug: #1733634 

This patch updates the SLO middleware and SegmentedIterable to add
support for user-specified inlined-data segments. Such segments will
contain base64-encoded data to be added before/after an object-backed
segment within an SLO. To accommodate the potential extra data we
increase the default SLO maximum manifest size from 2MiB to 8MiB.
The default maximum number of segments remains 1000, but this will
only be enforced for object-backed segments.
This patch is a prerequisite for a future patch enabling the
download of large objects as tarballs. The TLO patch will be added
as a dependent patch later.
UpgradeImpact
=============
During a rolling upgrade, an updated proxy may write a manifest that
out-of-date proxies will not be able to read. This will resolve itself
once the upgrade completes on all nodes.
Change-Id: Ib8dc216a84d370e6da7d6b819af79582b671d699