openstack/swift - swift - OpenDev: Free Software Needs Free Tools

We fixed swift-dispersion-report already; -populate needed the same fix
or else it'd hit a "maximum recursion depth exceeded" error.
Change-Id: I2d22e087a88c9e8003621feb26460ab6e5ce2a57
Related-Change: I24f4bcc3d62dc37fd9559032bfd25f5b15f98745
Closes-Bug: #1895346
Related-Bug: #1863680 

The current behavior is really painful when you've got hundreds of shard
ranges in a DB. The new summary with the states is default. Users can
add a -v/--verbose flag to see the old full detail view.
Change-Id: I0a7d65f64540f99514c52a70f9157ef060a8a892

Now that we can have null bytes in Swift paths, we need a way for
operators to be able to locate such containers and objects. Our usual
trick of making sure the name is properly quoted for the shell won't
suffice; running something like
 swift-get-nodes /etc/swift/container.ring.gz $'AUTH_test/0円versions0円container'
has the path get cut off after "AUTH_test/" because of how argv works.
So, add a new option, --quoted, to let operators indicate that they
already quoted the path.
Drive-bys:
 * If account, container, or object are explicitly blank, treat them
 as though they were not provided. This provides better errors when
 account is explicitly blank, for example.
 * If account, container, or object are not provided or explicitly
 blank, skip printing them. This resolves abiguities about things
 like objects whose name is actually "None".
 * When displaying account, container, and object, quote them (since
 they may contain newlines or other control characters).
Change-Id: I3d10e121b403de7533cc3671604bcbdecb02c795
Related-Change: If912f71d8b0d03369680374e8233da85d8d38f85
Closes-Bug: #1875734
Closes-Bug: #1875735
Closes-Bug: #1875736
Related-Bug: #1791302 

This commit reduce the number of I/O done by the swift-object-relinker.
First, it saves a progress state of relinking and cleanup in case the
process is interrupted during the operation. This allow to resume
operation without rescanning all partitions.
Secondly, it prevents from being scanned by relink and cleanup all
partitions that are bigger than 2^part_power (or (2^next_part_power)/2).
These partitions were not existing before the beginning of the part_power
increase, so there is nothing to relink or cleanup.
Thirdly, it reverse-orders the partitions to scan so that some useless
work is avoided. If a device contains partitions 1 and 3, relinking
partition 1 will create "new" objects in partition 3, that will need to
be scanned when the relinker will work on partition 3. It is useless. If
partition 3 is done first, it will only contain the objects that need to
be relinked.
Fourthly, it allows to specify a unique device to work on.
To do that, some hooks were added in audit_location_generator to allow
to execute some custom code before/after iterating a
device/partition/suffix/hash.
Change-Id: If1bf8ed9036fb0ec619b0d4f16061a81a1af2082

Change-Id: I1f140ae00cbd25b23c9a40ee91dccee8c7c15d81

"127.0.0.1s:sdas" is confusing at best.
Change-Id: I37f78d5993082ac29b001e9563aa4b24fd009a27

Previously, we'd get a KeyError trying to read headers.
Change-Id: I5d9f86784a3e39577ab010d29d8d03b26ffda357

Change-Id: I26c5fe9d45a9765da0d30138ea2df16fd4e73d57

Walking through the kernel logs backwards requires that we open them
in binary mode. Add a new option to allow users to specify which
encoding should be used to interpret those logs; default to the same
encoding that open() uses for its default.
Change-Id: Iae332bb58388b5521445e75beba6ee2e9f06bfa6
Closes-Bug: #1847955 

Change-Id: I7a2f19862817abf15e51463bd124293730451602

Change-Id: Id1280abd92e8bb02fcaa4701a0e9d211d9d6e33e

The previous locking method would leave the lock dir lying around
if the process died unexpectedly, preventing others swift-recon-cron
process from running sucessfuly and requiring a manual clean.
Change-Id: Icb328b2766057a2a4d126f63e2d6dfa5163dd223

Provides a simple, experimental, CLI tool to generate a
composite ring from a list of component builder files.
For example:
 swift-ring-composer <composite-file> compose \
 <builder-file> <builder-file> --output <ring-file>
Commands available:
- compose: compose a list of builder file to a composite ring
- show: show the metadata for a composite ring
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: I25a79e71c13af352e19e4358f60545265b51584f

The sharder daemon visits container dbs and when necessary executes
the sharding workflow on the db.
The workflow is, in overview:
- perform an audit of the container for sharding purposes.
- move any misplaced objects that do not belong in the container
 to their correct shard.
- move shard ranges from FOUND state to CREATED state by creating
 shard containers.
- move shard ranges from CREATED to CLEAVED state by cleaving objects
 to shard dbs and replicating those dbs. By default this is done in
 batches of 2 shard ranges per visit.
Additionally, when the auto_shard option is True (NOT yet recommeneded
in production), the sharder will identify shard ranges for containers
that have exceeded the threshold for sharding, and will also manage
the sharding and shrinking of shard containers.
The manage_shard_ranges tool provides a means to manually identify
shard ranges and merge them to a container in order to trigger
sharding. This is currently the recommended way to shard a container.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7f192209d4d5580f5a0aa6838f9f04e436cf6b1f

I'm not really clear on why a sqlite3.OperationalError should cause us to
retry with stale_reads_ok=True, but swift.common.exceptions.LockTimeout
*definitely* should.
Change-Id: I707dec1d11b8db80bc8fbee30662b319bf10d6a5

Similar to the object replicator and reconstructor, these arguments
are comma-separated lists of device names and partitions,
respectively, on which the account or container replicator will
operate. Other devices and partitions are ignored.
Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361

Bring under test
 - test/unit/cli/test_dispersion_report.py
 - test/unit/cli/test_info.py and
 - test/unit/cli/test_relinker.py
I've verified that swift-*-info (at least) behave reasonably under
py3, even swift-object-info when there's non-utf8 metadata on the
data/meta file.
Change-Id: Ifed4b8059337c395e56f5e9f8d939c34fe4ff8dd

Change-Id: I862b74e0d9b20ba149581c1add6473dc1e5b2859

Change-Id: I56dd9c646faba91e9f124f343ea0e08f8c3c4249

This has been deprecated since Swift 2.10.0 (Newton) including a
message that it would go away. Let's actually remove it.
Change-Id: I7d3659761c71119363ff2c0c750e37b4c6374a39
Related-Change: Ifa8bf636f20f82db4845b02d1b58699edaa39356

Change-Id: I98643d6acf248840a8360f31e446bc8ecb834898

...rather than only comparing the ETag from the last response over and
over again.
NB: This tool *does not* like EC data :-(
Change-Id: Idd37f94b07f607ab8a404dd986760361c39af029
Closes-Bug: 1266636

Add a --drop-prefixes flag to swift-account-info, swift-container-info,
and swift-object-info. This makes the output between the three more
consistent.
Change-Id: I98252ff74c4983eaad0a93d9a9fc527c74ffce68

Change-Id: I81736080fc478c2b69d5b71edd0cada39aad9400

This patch allows to import the dispersion report tool, and thus making
it more easily usable within other Python tools. This can be also used
in a follow up patch to add some tests for the report tool.
It also fixes a bug when using the "--dump-json" option - until now it
returned the policy name and made the JSON invalid.
Change-Id: Ie0d52a1a54fc152bb72cbb3f84dcc36a8dad972a

Clarify in usage statement and man pages that CLI override options for
swift-object-reconstructor and swift-object-replicator only have
effect when --once is used.
Also add a link to object reconstructor source code docs to the doc
index page for consistency with the other object services.
Change-Id: If348b340d59a672d3a19d4df231ebdb74f4aed51

If swift-recon/swift-get-nodes/swift-object-info is used with the
swiftdir option they will read rings from the given directory; however
they are still using /etc/swift/swift.conf to find the policies on the
current node.
This makes it impossible to maintain a local swift.conf copy (if you
don't have write access to /etc/swift) or check multiple clusters from
the same node.
Until now swift-recon was also not usable with storage policy aliases,
this patch fixes this as well.
Closes-Bug: 1577582
Closes-Bug: 1604707
Closes-Bug: 1617951
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: I13188d42ec19e32e4420739eacd1e5b454af2ae3

Change-Id: I510e05f4cf16cc6740f673c27cea6bc3899938c1

This patch adds methods to increase the partition power of an existing
object ring without downtime for the users using a 3-step process. Data
won't be moved to other nodes; objects using the new increased partition
power will be located on the same device and are hardlinked to avoid
data movement.
1. A new setting "next_part_power" will be added to the rings, and once
the proxy server reloaded the rings it will send this value to the
object servers on any write operation. Object servers will now create a
hard-link in the new location to the original DiskFile object. Already
existing data will be relinked using a new tool in the new locations
using hardlinks.
2. The actual partition power itself will be increased. Servers will now
use the new partition power to read from and write to. No longer
required hard links in the old object location have to be removed now by
the relinker tool; the relinker tool reads the next_part_power setting
to find object locations that need to be cleaned up.
3. The "next_part_power" flag will be removed.
This mostly implements the spec in [1]; however it's not using an
"epoch" as described there. The idea of the epoch was to store data
using different partition powers in their own namespace to avoid
conflicts with auditors and replicators as well as being able to abort
such an operation and just remove the new tree. This would require some
heavy change of the on-disk data layout, and other object-server
implementations would be required to adopt this scheme too.
Instead the object-replicator is now aware that there is a partition
power increase in progress and will skip replication of data in that
storage policy; the relinker tool should be simply run and afterwards
the partition power will be increased. This shouldn't take that much
time (it's only walking the filesystem and hardlinking); impact should
be low therefore. The relinker should be run on all storage nodes at the
same time in parallel to decrease the required time (though this is not
mandatory). Failures during relinking should not affect cluster
operations - relinking can be even aborted manually and restarted later.
Auditors are not quarantining objects written to a path with a different
partition power and therefore working as before (though they are reading
each object twice in the worst case before the no longer needed hard
links are removed).
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
[1] https://specs.openstack.org/openstack/swift-specs/specs/in_progress/
increasing_partition_power.html
Change-Id: I7d6371a04f5c1c4adbb8733a71f3c177ee5448bb

replace ring_name = basename(ring_path)[:len('ring.gz')]
with ring_name = basename(ring_path)[:-len('.ring.gz')]
Change-Id: I741e46d116c8fc7c2e91a51da4284302eec3aa41
Closes-bug: #1668736 

- Verify .ring.gz path exist if ring file is the first argument.
- Code Refactoring:
 - swift/cli/info.parse_get_node_args()
 - Respective test cases for info.parse_get_node_args()
Closes-Bug: #1539275
Change-Id: I0a41936d6b75c60336be76f8702fd616d74f1545
Signed-off-by: Sachin Patil <psachin@redhat.com>

Fixies this problem:
* swift-drive-audit needs to be run by root, because only root have
 "umount" permission
* swift-object servers typically runs as user swift
* if swift-drive-audit is run by root, /var/cache/swift/drive.recon is
 owned by root, with 0o600
* recon middleware (inside swift-object-server) can't read this cache
 file: swift-object: Error reading recon cache file
This patch adds "user" option to drive-audit config file. Recon cache
is chowned to this user.
Change-Id: Ibf20543ee690b7c5a37fabd1540fd5c0c7b638c9

swift-recon-cron looks at the drives mounted in directories below
/srv/node, but before this commit, it tried to call listdir() on
everything in this directory, even if it is not a directory.
Change-Id: Id281352f7ab6ecb520eb00f3649873d8c8678608
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>

F812 list comprehension redefines <variable> from line ...
While the current violations were benign, this sort of code can easily
lead to subtle bugs. Seems worth checking, especially given how cheap it
is to bring existing code in line with it.
Change-Id: Ibdcf9f93b85a1f1411198001df6bdbfa8f92d114

python-swiftclient includes an improved and tested method to generate
tempurls. The command syntax is essentially the same, therefore we can
deprecate this one by importing that method.
python-swiftclient is not added as a requirement; if the import fails
due to a missing swiftclient module it will just raise a deprecation
warning.
Closes-Bug: #1607523
Closes-Bug: #1607519
Change-Id: Ifa8bf636f20f82db4845b02d1b58699edaa39356

Change-Id: Icbb4c6d461ded4fab2afade09e718b3a74917717
Signed-off-by: Thiago da Silva <thiago@redhat.com>