openstack/ironic-python-agent - ironic-python-agent - OpenDev: Free Software Needs Free Tools

The original implementation of the skip block devices for RAID arrays:
https://review.opendev.org/c/openstack/ironic-python-agent/+/852999
introduced a couple bugs which were uncaught:
1. Key error when a holder disk contains just logical disks on the skip list.
2. RAID arrays on skip list throw "Failed to remove partitions" because they are not removed from the list of remaining RAID devices when running wipefs
3. list_block_devices_check_skip_list does not match volume names to RAID arrays
4. MD superblock wrongly checked (detail instead of examine)
5. Partition tables are being created when a partition is on a skip list
6. EFI partition handling in a scenario when a partition on the same physical disk is not deleted
Closes-bug: #2080871
Signed-off-by: Jakub Jelinek <jakub.jelinek@cern.ch>
Signed-off-by: Morten Stephansen <morten.kaastrup.stephansen@cern.ch>
Change-Id: I59b65c6b69af2385ed8a5dcd427e4d9c91f90abe

There is a conditional which is supposed to check whether there are
any erasable devices. However, in the current state, the conditional
is wrong as the call is missing the node as a parameter.
Signed-off-by: Jakub Jelinek <jakub.jelinek@cern.ch>
Signed-off-by: Morten Stephansen <morten.kaastrup.stephansen@cern.ch>
Change-Id: I38768b9ba3dc1bb5160e5841865450a8d7df5466

In test_create_configuration_with_different_disks_skip_list there are redundant side-effects which are never used.
Change-Id: I7b6acbd6583831ba82d6a68ce0d8410ee0bd18a2
Signed-off-by: Morten Stephansen <morten.kaastrup.stephansen@cern.ch>

Some motherboards return UTF-8 for the efibootmgr. This results in IPA not being able to remove duplicate records, etc.
This change implements a check for UTF-16 compatibility and tries to decode efibootmgr output as UTF-8 otherwise.
Closes-bug: #2072336
Co-Authored-By: Jakub Jelinek <jakub.jelinek@cern.ch>
Change-Id: I35432773826d13edb1dc9dd25f99bb0907a8fa0d
Signed-off-by: Morten Stephansen <morten.kaastrup.stephansen@cern.ch>
Signed-off-by: Jakub Jelinek <jakub.jelinek@cern.ch>

Adds a tran field to the block device and allow to use it
as a root device hint.
Change-Id: I3fc83730a6100abb2b2aa98fc894713ecbbe3043
Closes-Bug: #2100951
Signed-off-by: Kaifeng Wang <kaifeng.w@gmail.com>

Adds a wall timeout `image_download_max_timeout` to enforce an upper
bound on total download duration.
While the per-chunk timeout protects against stalled reads, downloads
that trickle in just under the timeout threshold (e.g., due to heavy
TCP retransmits) can hang for longer than intended.
Now, if the total allowed time is exceeded, the download is aborted with
a non-retryable `ImageDownloadTimeoutError` regardless of per-chunk
retry or connection success.
A value of 0 (the default) disables this feature.
Closes-Bug: #2115995
Change-Id: I3b56d21abae0488853bfed14072ba21116d47baf
Signed-off-by: Afonne-CID <afonnepaulc@gmail.com>

Returning None means "exclude any device". It still works if the only
hardware manager is GenericHardwareManager, but enabling any more
without overriding filter_device causes all devices to get filtered.
Change-Id: I25eb028baa1b9182caee07a0b935e9fa107999a8
Closes-Bug: #2117234
Signed-off-by: Dmitry Tantsur <dtantsur@protonmail.com>

Create a file efibootmgr with the verbose output of the efibootmgr command
when collecting the system logs. This can be used for debugging of boot order.
Change-Id: Ic957024d19bb01a45a3014dc2a5e4492d087e893
Signed-off-by: Jakub Jelinek <vilouskubajj@gmail.com>

Change-Id: Iee7f22e9958b68f263d267c434678478b9a38648
Signed-off-by: Dmitry Tantsur <dtantsur@protonmail.com>

Change-Id: I1eb8c9637c38496b5d4dd6493be43f414f35680c
Signed-off-by: Dmitry Tantsur <dtantsur@protonmail.com>

... instead of using oslo.service. Current usage of oslo.service is
too limited to add the dependency, because
 - oslo.service registers multiple options but only two of these are
 used
 - the wrap implementation from oslo.service is not actually used
Change-Id: I4e8f18951d73e329a54cf6546344c5704fe4aa90
Signed-off-by: Takashi Kajinami <kajinamit@oss.nttdata.com>

My use case for this feature is to exclude network devices that use
the cdc_ether driver. These USB network interfaces often cause all sorts
of issues. For example, some models have the same hardcoded MAC address,
which breaks inspection.
Currently, to exclude a certain device, a hardware manager must override
the entire listing function (in my case, list_interfaces). Not only is
it tedious, but it also requires constantly updating the hardware
managers to match the implementation in GenericHardware. Realistically,
it will cause hardware manager authors to inherit GenericHardware, which
is the opposite of how hardware managers should be written.
Note that the node-level skip list only affects root device selection
and cleaning for block devices. This feature affects everything that
uses list_block_devices and is applied before the node-level skip list.
This change adds a new hardware manager call filter_device. For each
network, block or USB device, it allows a hardware manager to do either
of four things:
1. Delegate the decision to a lower level hardware manager by raising
 IncompatibleHardwareMethodError
2. Remove the device by returning None
3. Change the device by returning a modified instance
4. Return the device unchanged to keep it in the listing.
Note that I'm removing debug logging when IncompatibleHardwareMethodError
is raised. Not only the log message is incorrect (the error does not
necessarily mean that the method is not implemented at all), it already
noticeable space in the logs, and with this change will become very
noisy.
Change-Id: I5437343af6c6157882bcf0600dd89bd20478c948
Signed-off-by: Dmitry Tantsur <dtantsur@protonmail.com>

The current code in GenericHardware.evaluate_hardware_support ends up
using hardware manager calls, which then use partly initialized hardware
manager list and can even cause a recursion.
This change introduces a new optional call initialize() which is
guaranteed to run:
1) After all hardware managers have been evaluated
2) After the hardware manager cache is populated
3) In the order of the support level of hardware managers
Change-Id: I068d3d73483c161062aa3b48f3154a2d99941382
Signed-off-by: Dmitry Tantsur <dtantsur@protonmail.com>

These objects are frequently logged or compared in unit tests.
It's very helpful to be able to inspect their content.
Change-Id: Ib725dcd5f54f4492205f95974d887b8b42c74039

When creating multiple software RAID logical disks that use different
sets of physical devices, the partition indices were incorrectly shared
across all devices. This caused the second RAID array creation to fail
because it tried to use partition indices that didn't exist on those
specific devices.
This change fixes the issue by tracking partition indices separately for
each physical device, ensuring that each device's partitions are numbered
correctly starting from their first available index.
Closes-Bug: #2115211
Change-Id: I440db4654f3d1d54274d1eee8c4b21c2b0a18d22
Signed-off-by: Mohammed Naser <mnaser@vexxhost.com>

Fetching the permanent MAC address of the interface instead of the
default one allows to get the right one in case it got changed during
setup (likely with a bonding setup).
In order to fetch the permanent MAC address of a given interface, one
can either use Netlink (either rtnetlink or ethtool), or use ethtool
ioctl.
The use of ioctl feels simpler and requires no additional dependency.
The implementation falls back to older behavior should an error occur.
Closes-Bug: #2103450
Change-Id: I54151990e396ddcf775128ca24d3db08e45c256d
Signed-off-by: Nicolas Belouin <nicolas.belouin@suse.com>

This change removes several usages of eventlet from IPA:
- Upgrades all requirements on oslo library versions to new ones that
 support non-eventlet use.
- Removes use of the eventlet wsgi server (via oslo_service.wsgi) and
 replaces it with the cheroot wsgi server.
- Removes explicit patching of python modules with eventlet
Note that due to some oslo libraries still using ``eventlet`` to detect
and workaround it's use. This means that it is still installed in
environments alongside IPA, even if it's not used or patched into any
modules.
Depends-On: https://review.opendev.org/c/openstack/requirements/+/947727
Change-Id: I9accab2d5e9529a88ef5d3db85e76901f14114eb

These files are not actually executable.
Change-Id: Id4208a91c005b8199d62320882c4f14dcd7f7761

Change-Id: Id85563015b3ea9e2802baa7b8ab1ca1d858568d5

- Implemented manual cleaning
- Refactored the code
Change-Id: I301aaf9dfd6aff90f505148b65e75033f5043553

Update container-based cleaning hardware manager to use ironic
conductor config.
Note:
- Moved conf variables from __init__ and evaluate_hardware_support
 as the config overwritten after those process
- Utilized getattr instead of making methods beforehand. The methods
 created don't stick for a new instance.
Partial-Bug: #2100556
Change-Id: I53d5a4f112fbed455d5574840611ef6ea2db3eae

- Python 3.13 uses time.time_ns for logging
https://github.com/python/cpython/blob/main/Lib/logging/__init__.py#L303
Change-Id: I3de44cc0fda662f3d5b1c6ea8add973cf2ed3bd9

Change-Id: I54dc23d8e8cec12a4685a82c4807ddd6a2267533

Eventlet patches two things in socket, effectively:
- create_connection
- various greendns things
By adding this environment variable, we're going to disable a large portion of the greened module code in IPA, which will be a boost to our migration off.
Change-Id: I1f94238c8d83f9e7cb0f7e096172ffb7c20c862b

Implement container-based cleaning process
Partial-Bug: #2100556
Change-Id: I39b92462d1454df888fc413e0aac439b9df199f7

Change-Id: I916fe8187fb9d8513852b620df133309fbc92af1

Using prlimits is incompatible with passing arguments as a list:
oslo.concurrency ends up executing something like:
/opt/ironic-python-agent/bin/python3 -m oslo_concurrency.prlimit \
 --as=2147483648 -- ['env', 'LC_ALL=C', 'LANG=C', 'qemu-img', 'info', \
 '/tmp/cirros-0.6.2-x86_64-disk.img', '--output=json']
Which obviously fails. I don't understand how our CI has worked so far,
but the Metal3 BMO suite fails on this.
Change-Id: I46dbcb0f73bcbe09bb89b5c7195259570412698e

To help ease upgrades to Victoria, IPA had a knob added
to enable operators to express if agent tokens were required
in their deployment. Since then, the feature is required, however
we left the logic enabling the fun upgrade case handling.
At this point, this knob serves no further use, and can be removed.
Change-Id: I202f06e1b6598a802c9853fb99201c55e7a40cb1