openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Walking through the kernel logs backwards requires that we open them
in binary mode. Add a new option to allow users to specify which
encoding should be used to interpret those logs; default to the same
encoding that open() uses for its default.
Change-Id: Iae332bb58388b5521445e75beba6ee2e9f06bfa6
Closes-Bug: #1847955 

Fixies this problem:
* swift-drive-audit needs to be run by root, because only root have
 "umount" permission
* swift-object servers typically runs as user swift
* if swift-drive-audit is run by root, /var/cache/swift/drive.recon is
 owned by root, with 0o600
* recon middleware (inside swift-object-server) can't read this cache
 file: swift-object: Error reading recon cache file
This patch adds "user" option to drive-audit config file. Recon cache
is chowned to this user.
Change-Id: Ibf20543ee690b7c5a37fabd1540fd5c0c7b638c9

As much as anything, I'm just tired of seeing a bunch or piecemeal
fixes.
Note that we *need* to include
 from __future__ import print_function
in order to support things like
 print() # Would print "()" (the repr of an empty tuple) otherwise
 print(foo, end='') # Would SyntaxError
 print(bar, file=sys.stderr) # Would SyntaxError
Change-Id: I8fdf0740e292eb1ee785512d02e8c552781dcae1

An additional info log message was added for case of
running drive-audit without failed device unmounting.
Change-Id: I11abee40a712b6c6de65e63626b6f7f0a9c9f4c7

Changes Of py3 in bin :
* https://review.openstack.org/#/c/196835/
* ConfigParser from six.moves
Change-Id: Ic0374c8e09dfd595ec12c4d31b17dad30eaa803c

This is a follow-on from a previous commit which added recon info
for swift-drive-audit (https://review.openstack.org/#/c/122468/).
Here, the "--drievaudit" option is added to swift-recon tool. This
feature gives the statistics for the system-wide drive errors flagged
by swift-drive-audit. An example of the output is as follows:
(verbose mode)
swift-recon --driveaudit -v
===============================================================================
--> Starting reconnaissance on 5 hosts
===============================================================================
[2015年03月11日 17:13:39] Checking drive-audit errors
-> http://1.2.3.4:6000/recon/driveaudit: {'drive_audit_errors': 14}
-> http://1.2.3.5:6000/recon/driveaudit: {'drive_audit_errors': 0}
-> http://1.2.3.6:6000/recon/driveaudit: {'drive_audit_errors': 37}
-> http://1.2.3.7:6000/recon/driveaudit: {'drive_audit_errors': 101}
-> http://1.2.3.8:6000/recon/driveaudit: {'drive_audit_errors': 0}
[drive_audit_errors] low: 0, high: 101, avg: 30.4, total: 152, Failed: 0.0%, no_result: 0, reported: 5
===============================================================================
Change-Id: Ia16c52a9d613eeb3de1a5a428d88dd1233631912

The drive-audit detects error log about a device and comments out it
in /etc/fstab. When the error log is generated several times, it
comments out the line for each time.
This patch makes drive-audit to check if the device is already
commented out, and prevents redundant commenting out.
Change-Id: Ia542d35b58552dde0f324bb9c42531f98c9058fa

This patch adds console logging ability to swift-drive-audit.
There are cases where logging to console is necessary when drive-audit
is done. This can be consumed for flagging errors in monitoring tools
such as icinga.
DocImpact
Change-Id: Ia1e1effcbd89bd2cf6d5b8c64019f1647c736a3a

Closes-Bug: #1379767
Change-Id: Ie8ec5e9ffb977ab39aedeb878e6aae9f0b62530b

This patch adds two new features to swift-drive-audit. The first
is an option in the drive-audit.conf file that allows the operator
to prevent the drives ever being unmounted automatically,
regardless of the amount of errors present. This could be of
benefit in very small systems consisting of only one or two drives
where the operator would like to manually unmount/fix the
particular drive(s) and minimise any potential downtime.
The second is another option in drive-audit.conf that allows the
operator to select a recon directory. This directory will then
have a drive.recon file which will keep an up-to-date record of
the swift drives and any errors associated with them. An example
of the output would be as follows:
{"/srv/node/disk2": "0", "/srv/node/disk3": "25", "/srv/node/disk0": "0",
"/srv/node/disk1": "0", "/srv/node/disk10": "0", "/srv/node/disk7": "0",
"/srv/node/disk4": "137", "/srv/node/disk5": "0", "/srv/node/disk8": "0",
"/srv/node/disk9": "0", "/srv/node/disk6": "0", "/srv/node/disk11": "60"}
This would allow the operator to monitor the errors on the swift
drives without having to spend time searching through logs. Also, if
this is accepted, it should be possible to add an option to
swift-recon that would keep track of this at a system level.
Change-Id: Ib5dacf8622b7363e070c274c7c30c8ead448a055

We want to show log files in descending order.
Change-Id: I6f79d9f5a73afa43efec36c3ca99afcde3fe7813
Closes-Bug: #1311405 

Fix misspellings detected by:
* pip install misspellings
* git ls-files | grep -v locale | misspellings -f -
Change-Id: I6594fc4ca5ae10bd30eac8a2f2493a376adcadee
Closes-Bug: #1257295 

Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176 

Making it possible for one to overwrite the default set of regexes
used to search for device block errors in the log file. Also making
the log file naming pattern configurable by setting them in the
drive-audit.conf file.
Updating "Detecting Failed Drives" section on the admin guide as well.
Change-Id: I7bd3acffed196da3e09db4c9dcbb48a20bdd1cf0

Change-Id: I3d4a31111c3044da06611405ce80f208ef8a0ce3

Sometimes there is no date at the beginning of a line in kern.log.
Although it does not happen often, there should be a check ensuring
the program doesn't crash in case it happens.
Added try-except block surrounding parsing string to date format.
Change-Id: I44a101266582eea2199189a006afa1037a9bd4ea
Fixes: bug #1152658 

Change supports kern.log rotation in order to avoid loss
of significant information.
There is a year change functionality added as kern.log
does not keep record of year.
There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it
Fixes Bug 1080682
Change-Id: I93436c405aff5625396514000cab774b66022dd0

This required a bunch of whitespace-poking of the scripts in bin, but
that's all. Now every file in swift/ and bin/ is pep8-1.3.3-compliant,
so hopefully we can be done with this pep8 stuff for a good long time.
Change-Id: I44fdb41d219c57400a4c396ab7eb0ffa9dcd8db8

Fixes Bug 1049081
Change-Id: If977080350cc5cdb6bc633b6af7d3c490ed23d46

Also made tox's PEP8 check look at the scripts in bin/ to keep them
PEP8-compliant.
Change-Id: I710365ea929d7fc15578d5f742a236bad47ef28e

Change-Id: Ifd909d3561c2647770a7e0caa3cd91acd1b4f298

addresses bug 827913
Change-Id: I691eee191f5951186158c553281f88aae9e5d25f

as a deb install.