openstack/ironic-python-agent - ironic-python-agent - OpenDev: Free Software Needs Free Tools

Change-Id: I2f0e581575112d6c7ba0d211661cab3e0b6caca6

The two functions work_on_disk and create_config_drive_partition contain
a substantial part of the deployment logic. Previously we placed them in
ironic-lib for re-using on the conductor side in the iSCSI deploy
interface. Since the iSCSI deploy is going away, we can move this code
to ironic-python-agent to simplify maintenance.
Imports code from ironic_lib commit 9fb5be348202f4854a455cd08f400ae12b99e1f2.
Change-Id: I6cbcd81533f135208b57746cb0e33ffdfaf94eee

Due to a regression in lshw introduced by
https://github.com/lyonel/lshw/pull/60, there are some versions in the
wild that do not return sizes for memory banks <32GiB. In those cases,
work around the problem by looking at the top-level size (if available)
to find the total size. Previously we assumed that we only needed the
top-level size when there was no list of memory banks.
The issue is fixed upstream by https://github.com/lyonel/lshw/pull/65,
but the erroneous patch is still present in the lshw-B.02.19.2-5.el8
package in CentOS 8.4 and 8.5.
Change-Id: I6eb5981d28b9ae368239af0c1d0ec32ff79d95b3
Story: #2008865
Task: 42395

Change-Id: I5d08deed86d79a7ea0b7a1625122af595037dab5

Pbr is a very heavy package to depend on. It requires git-core, which is
16 MiB on my Fedora. We only use it to detect the version, which can be
done without pbr using a much lighter importlib_metadata.
Copied from https://review.opendev.org/c/openstack/osprofiler/+/739379
Change-Id: I5f434e6bfde6f645804941f3a36d5458a28270e7

After GPT and MBR are destroyed systemd-udevd gets triggered
which may hold /dev/sda open preventing qemu-img from writting
its image.
Story: 2008830
Task: 42312
Change-Id: I6105192a16fcb7f6898910e8d0ab824d731d491d

The line we're looking for is not there when IPA is in a container, at least
for CentOS based containers. Just fall back to sysrq on errors.
Change-Id: Ie4ee605ad9c6cda58808512a563247175859c71e

Currently if one interface cannot be handled (e.g. it has empty MAC),
the whole collection fails. Ignore unsupported interfaces instead.
Change-Id: Ibdaad62b39c239d4f3fb3111c2fae9e31e877b28

_early_log prints to stdout, which is fine in some cases,
however in other cases it gets lost in the shuffle of process
launch by things like systemd.
Lets try to save everything, and re-log it so it is easy to
debug early issues.
Change-Id: I334a9073d17cccec4c669fae82edc3e388debc5c

Follow-up to 8dd6589e66: PATH is not a
valid lsblk tag, we need to use KNAME with -p flag.
Also add a vmedia job to avoid breakages in the future. It's added
non-voting because we have a deadlock with this change:
https://review.opendev.org/c/openstack/ironic/+/783722
Change-Id: Ifffeac9c1c4d394526d655eaa14c9fe7bd3a1e5e

Virtual media devices based logic needs to be
guarded from being used or considered based upon
if the machine actually booted from virtual media,
or not.
At the same time, actual devices need to be checked
in order to make sure they align with what we expect
in order to prevent consideration of content which
should not be leveraged.
Change-Id: If2d5c6f4815c9e42798a2d96d59015e1b1dbd457
Story: 2008749
Task: 42108

IPA standalone mode is a developer-only option, and if enabled
accidentally on a production agent could cause undesired behavior.
Developers who need this behavior should build a purpose-built agent,
with standalone hardcoded to True in cmd/agent.py.
Change-Id: Icc67dbe15acbbf6fee886f274d2169a0769a5053

Change-Id: Ifb75a5f6f01bd98011464eb05f98d8db001dcd54

Change-Id: I0b597ddbc71c133abe6c0acfd8f49e3af4e896bb

The root UUID changes after a streamed partition image is written to
the block device, causing later deployment failure when assuming the
old UUID.
This change updates the root UUID after streaming the partition image
is complete.
This issue may have been missed in local testing because deploying the
same image repeatedly will result in stable root UUID across runs.
Change-Id: Ice4630c16fc216980488d1427f3b02e1b8a417fa

For some drives, the partition e.g. `/dev/sda1` will not have the
'ro' file which can result in a metadata erasure failure but the base
device (`/dev/sda`) will have this file. Add an additional check
for the base device.
Change-Id: Ia01bdbf82cee6ce15fabdc42f9c23036df55b4c5
Story: 2008696
Task: 42004

The param check_exit_code from the processutils extension execute has
default already at [0]
See:
https://opendev.org/openstack/oslo.concurrency/src/branch/master/oslo_concurrency/processutils.py#L214
Change-Id: Iedff5325e0737556d5eb3da601c984ddfc633873

We appear to be bumping up against this limit when deploying
RHCOS images(currently 977MB). Curiously the problem isn't
happening all the time but increasing the limit eliminates it.
This limit was intruduced to guard against a malicious image
allocating an arbitrary amount of memory. Nothing else runs
on hosts when IPA is running so we should be ok bumping up
the limit.
Story: #2008667
Task: #41955
Change-Id: I9405995915a874b00b7177c9642c5469d05d66a8

This change adds '-f' flag to nvme-cli calls during NVMe Secure Erase.
This removes nvme-cli output warning that the device is about to be
irreversibly deleted as well as the related 10 second delay which is
pointlessly increasing NVMe cleaning time.
Story: 2008290
Change-Id: I7b7b8b7d4f643b07d5c9dcf7ec35cf7ebedf44d1

At the moment, it is not possible for Ironic to clean up a
RAID array that is built from an entire device. This patch
allows it to do so by overriding the behaviour of attempting
to find the device name if the device names does not end with
a number and is a real block device.
Story: #2008663
Task: #41948
Change-Id: I66b0990acaec45b1635795563987b99f9fa04ac7

Also adapt unit tests
Change-Id: I37d050877daabc9dc0a5821cf20a689652b26f34

This change adds support for utilising NVMe specific cleaning tools
on supported devices. This will remove the neccessity of using shred to
securely delete the contents of a NVMe drive and enable using nvme-cli
tools instead, improving cleaning performance and reducing wear on the device.
Story: 2008290
Task: 41168
Change-Id: I2f63db9b739e53699bd5f164b79640927bf757d7

To reduce size of the hardware module and separate the raid specific
code in raid_utils, we move some functions and adapt the tests.
Change-Id: I73f6cf118575b627e66727d88d5567377c1999a0

IPA is not properly checking if the root partition is already
mounted. Device is being passed to os.path.ismount() instead
of the mount point.
Story: 2008631
Task: 41839
Change-Id: I37a6e7e6bbe0bbbb0317c6e55bb822dafe7cce20

This change adds a deploy step inject_files that adds a flexible
way to inject files into the instance.
Change-Id: I0e70a2cbc13744195c9493a48662e465ec010dbe
Story: #2008611
Task: #41794 

They take a lot of space and tehy're mainly static data.
Move them to a separate module that can also be shared with other tests.
Change-Id: I9c76c014430d524da7fa331c922976d283b870c3

It's somewhat confusing at the moment, since we're trying to find
a UEFI partition by UUID "None". Don't search for partition if
we don't know its UUID, and provide a better error message.
Change-Id: Ief874084132797a445ddae8009264712a05facfd

Adjusted unit tests accordingly.
Also removed redundant parenthesis.
Change-Id: I8e2cac5172f009d5204f83bd83e1f27cfd721f09

When running IPA unit tests on machines configured
with UEFI and GPT partition table, some tests will
fail.
Change-Id: I155a47242b526b8f243a5e94bc14da8431f1ab91

Diskimage-builder installs grub with option '--removable'[1], thus for
aarch64 no 'grubaa64.efi' file in efi directory only got 'BOOTAA64.EFI':
linaro@bm-ubuntu:~$ tree /boot/efi
/boot/efi
└── EFI
 └── BOOT
 └── BOOTAA64.EFI
2 directories, 1 file
[1]: 8f12d9530e/diskimage_builder/elements/bootloader/finalise.d/50-bootloader (L158)
Task: #41698
Story: #2008560
Change-Id: I9fc55c068ea980beae273411db9d3568eec25eb8

The kernel parameter lldp-timeout was deprecated removed in this patch.
Change-Id: I98da49e61d9ed3236cc495d1ab351eba0931473b

Previously, partition images were hard coded to be bios based
as opposed to consulting all of the values AND the node itself
before making the most appropriate determination. Now the agent
utilises the internal helper to properly determine the boot
mode when calling ironic-lib.
Story: 2008070
Task: 41265
Change-Id: Id5eeda69d5b9de2b393af414472d57b0d4380c43

The partition image support has been telling ironic-lib
that the machine will be local booted. While this is likely
harmless, and doesn't seem to break anythign, we should have
it match moving forward just to be on the safe side so we don't
accidently break things down the road.
Change-Id: I33e5d583964ef8c21aa04d7427bcd3957b89d449