openstack/ironic-python-agent - ironic-python-agent - OpenDev: Free Software Needs Free Tools

In order to make sure we have the correct time early, e.g.
by the time we create a TLS certificate, this patch proposes
to force an immediate NTP update when using chronyd. While
the previous approach uses the passed NTP server as well, the
update may happen only after chronyd has performed measurements
(which may be too late).
Story: #2009058
Task: #42843
Change-Id: I6edafe8edeb8549f324959e7a1ec175c3049a515

Add a clean step for network burn-in via fio. Get basic
run parameters from the node's driver_info.
Story: #2007523
Task: #42385
Change-Id: I2861696740b2de9ec38f7e9fc2c5e448c009d0bf

While investigating another grub issue, I was confused by the path
taken in the logs reported, and noticed that on a ramdisk, we might
not actually have a valid response to os.path.ismount, I'm guessing
depending on what in memory filesystem is in use while also coupled
with attempting to check a filesystem.
Adds a test to validate that exceptions raised on these commands
where this issue can be encountered, are properly bypassed, and also
adds additional logging to make it easier to figure out what is
going on in the entire bootloader setup sequence.
Change-Id: Ibd3060bef2e56468ada6b1a5c1cc1632a42803c3

Check if the ESP is already mounted before attempting to mount it
for the bootloader installation.
Change-Id: Ifd738b2c5663f1a211d7e13b5ba386be631d8db1

The IPA sends heartbeats to the conductor periodically and when
requested, e.g. at the end of asynchronous commands. In order
to avoid to send such notifications in too quick succession,
e.g. when two asynchronous commands finish at the same time or
when the periodic heartbeat was just sent right before a command
ended, this patch proposes to coalesce heartbeats which are
close together timewise and send only one for all of them
in a time interval of 5 seconds.
Co-Authored-By: Arne Wiebalck <arne.wiebalck@cern.ch>
Story: #2008983
Task: 42633
Change-Id: Idfbce44065e1e5a8b730b94741b2604c51f0ab14

It's not uncommon that some commands fail when collecting logs.
We already log all failures in utils.execute, no need to duplicate
them with a non-fatal ERROR logging.
Change-Id: If151b3a3be979bd2b3ce01030e5d6242ad74eaa3

As discussed during the upstream ironic community meeting on
Monday Dec 14 2020, the lower-constraints job is being removed.
Change-Id: I9392117ccdef8fc6ff3d95d667107d752459cb24

Adds support to identify and utilize a CSV file to signal which
bootloader to utilize, and set it when the OS is running as opposed
to when EFI is running. This works around EFI loader potentially
crashing some vendors hardware types when entry stored in the
image does not match the EFI loader record which was utilzied to
boot.
Grub2+shim specifically specifically needs the CSV file name
and entry label to match what the system was booted with in order
to prevent the machine from potentially crashing.
See https://storyboard.openstack.org/#!/story/2008962
and https://bugzilla.redhat.com/show_bug.cgi?id=1966129#c37
for more information.
Change-Id: Ibf1ef4fe0764c0a6f1a39cb7eebc23ecc0ee177d
Story: 2008962
Task: 42598
Co-Authored-By: Bob Fournier <bfournie@redhat.com>

Recent releases of redhat grub2 will always fail when installing to
EFI paths, to encourage a transition to the signed shim bootloader.
Partition image deploys avoid calling grub2-install with the
preserve-efi-assets functions. Deploying whole disk images doesn't
require grub2-install. This leaves whole disk images installed onto
softraid devices, which still attempts to call grub2-install.
This change will still attempt to run grub2-install in this
one remaining case, but will ignore any failure.
A future enhancement can avoid calling grub2-install entirely so that
non-redhat secure-boot capable images can keep their signed
bootloaders.
Story: 2008923
Task: 42521
Change-Id: If432ef795d64d76442d739eb4f7d155ff847041e

The patch bumps min version of tox to 3.18.0 in order to
replace tox's whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020年07月23日
Change-Id: I434a6330defbc5cc65788f4d64b7f2d4d2b37db7

Change-Id: I6a52b55237225f169ff59ac8854f739d9c0f92c7

We're currently requiring it twice: in image_info and in a separate
configdrive argument. I think we should eventually settle on separate
arguments for separate entities, so this change makes the value in
image_info optional with a goal to stop accepting it.
We could probably just remove the handling in image_info, but a
deprecation is safer.
The (unused in ironic) cache_image call is updated with an optional
configdrive arguments.
Story: #2008904
Task: #42480
Change-Id: I1e2efa28efa3ea7e389774cb7633d916757bc6ed

Change-Id: If890baf3545cff6cef7c645c42e7f9d9038c9aa7

qemu-img attempts to launch multiple threads by default *and*
attempts to have multiple memory allocation arenas to operate
from. While multithreading can be good for performance, this
pattern and the memory footprint for process launch and
dependencies can turn the memory footprint for a cirros image
conversion (16MB) into 1.2GB of memory being asked for by the
qemu-img tool.
In order to limit this impact, as the default number of arenas
is governed by the number of CPUs times the number 8, it seems
reasonable to lower this to a more reasonable number which
also helps keep our possible memory footprint from being exceeded.
Change-Id: I71a28ec59ec31c691205eb34d9fcab63a2ccb682
Story: 2008928
Task: 42528

To make this function useful for purposes other than efibootmgr
entries, this change moves the path manipulation to _run_efibootmgr.
This change also adds boot*.efi entries to BOOTLOADERS_EFI so that it
includes every entry in the UEFI Spec 2.9[1] Table 3-2 UEFI Image
Types.
[1] https://uefi.org/sites/default/files/resources/UEFI_Spec_2_9_2021_03_18.pdf
Story: 2008923
Task: 42521
Change-Id: Ibe02786609aa0de65115897d8f4a9b4f36c8aed2

Add a clean step for disk burn-in via fio. Get basic
run parameters from the node's driver_info.
Story: #2007523
Task: #42384
Change-Id: I5f5e336bd629846b3d779fd0fc7a2060b385b035

Change-Id: I0caa65561948f4e0934943a7a0d3a209701b5a59

Per documentation it improves performance when using -O host_device.
Change-Id: Ic6a97af9f865d07c9cb4257397a320475a28f88b

The command params can be huge when configdrive is used. There is no
point in sending them back, Ironic does not use them anyhow.
Story: #2008904
Task: #42479
Change-Id: I6e3db5db2042ca3fb5dafacfacf036fd7fc2fc4c

Missed in commit 24951b1029.
Change-Id: Iad5e8f161ac69b96b9332d83fe22b5e0b9192258

Change-Id: I2f0e581575112d6c7ba0d211661cab3e0b6caca6

The _manage_uefi code has a check where it attempts to just
identify the precise partition number of the device, in order
for configuration to be parsed and passed. However, the same code
did not handle the existence of a `p1` partition instead of just a
partition #1. This is because the device naming format is different
with NVMe and Software RAID.
Likely, this wasn't an issue with software raid due to how complex the
code interaction is, but the docs also indicate to use only whole disk
images in that case.
This patch was pulled down my one RH's professional services folks
who has confirmed it does indeed fix the issue at hand. This is noted
as a public comment on the Red Hat bugzilla.
https://bugzilla.redhat.com/show_bug.cgi?id=1954096
Story: 2008881
Task: 42426
Related: rhbz#1954096
Change-Id: Ie3bd49add9a57fabbcdcbae4b73309066b620d02

Setuptools v54.1.0 introduces a warning that the use of dash-separated
options in 'setup.cfg' will not be supported in a future version [1].
Get ahead of the issue by replacing the dashes with underscores. Without
this, we see 'UserWarning' messages like the following on new enough
versions of setuptools:
 UserWarning: Usage of dash-separated 'description-file' will not be
 supported in future versions. Please use the underscore name
 'description_file' instead
[1] https://github.com/pypa/setuptools/commit/a2e9ae4cb
Change-Id: I5b71e5ef287a9557ed60bd23cd97dfe74571b457

The two functions work_on_disk and create_config_drive_partition contain
a substantial part of the deployment logic. Previously we placed them in
ironic-lib for re-using on the conductor side in the iSCSI deploy
interface. Since the iSCSI deploy is going away, we can move this code
to ironic-python-agent to simplify maintenance.
Imports code from ironic_lib commit 9fb5be348202f4854a455cd08f400ae12b99e1f2.
Change-Id: I6cbcd81533f135208b57746cb0e33ffdfaf94eee

Add a clean step for memory burn-in via stress-ng. Get basic
run parameters from the node's driver_info.
Story: #2007523
Task: #42383
Change-Id: I33a83968c9f87cf795ec7ec922bce98b52c5181c

Add a clean step for CPU burn-in via stress-ng. Get basic
run parameters from the node's driver_info.
Story: #2007523
Task: #42382
Change-Id: I14fd4164991fb94263757244f716b6bfe8edf875

Remove the jobs that are now redundant.
Depends-On: https://review.opendev.org/c/openstack/diskimage-builder/+/788972
Depends-On: https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/788971
Change-Id: Ic0b7bc509652c23cec85e685d66de368fdc4761d