openstack/ironic-python-agent - ironic-python-agent - OpenDev: Free Software Needs Free Tools

Change-Id: Ia21690955b25b233a7b2e861abb5ac02653099f6
Depends-On: https://review.opendev.org/688911 

Depends-On: https://review.opendev.org/678884
Change-Id: I50a9f9dc15f3a782aaf4b8c4096a418136f4ebd4

Change-Id: Id9737067c678d4d01b20eae0b4a681c6cf9171a8

Change-Id: I56f7c9abdae1f406de1e086fdc0cac8bebca0e3c
Task: 33745
Story: 2005114

Change-Id: I4cefe044e0a0ee727e2c3dfcf1c873c72cabc8a0
Story: 2005886
Task: 33735

With the addition of Software RAID we need 'mdadm' in our images.
Change-Id: I29a4ce98dbd456b104683a2202ba3115b966469d

Change-Id: I7473b96d47d959fc4c02cecbfaf4785d5f92996a

We use the url from tox.ini to download the upper constraints and
recently that has changed to a redirect and curl doesn't
correctly follows redirects by default but stops at the first
page.
Adding -L option to curl command makes curl follows the redirect
until the final location is found and should allow the correct
file to be downloaded.
Change-Id: I25c724ffe189d414643c910be0ef61c550f802b7

The Makefile's command to clean only the tinyipa ramdisk build
is `make clean_build`, adjust the name according to the Makefile.
Change-Id: Icb9f2b7213de4be1eda71d92150f76e5efd72887

Tinycore is the base image for testing Ironic Python Agent.
In the effort of moving toward a full python3 friendly
environment, with this patch we start building the tinyipa
image with python3 by default.
This patch is a Follow-up to
https://review.openstack.org/#/c/635487/
Change-Id: Id446a67a31d25a21b086f85b459c0bec6b58ca7e

This patch uses ``USE_PYTHON3`` variable to build
Ironic Python Agent with python 3 inside a tinycore
image.
It modifies the content of ``buildreqs.lst`` and
``finalreqs.lst``, adding specific package files
based on the python version used to build IPA.
Since we're using python 3.6, bytecode optimized
files have .pyc extension and are located under a
__pycache__ dir [1]; as an optimization option,to
reduce the size of the final image, all bytecode
generated files will be moved to the original
source location and the source code will be
removed.
[1] https://www.python.org/dev/peps/pep-0488/
Change-Id: I3448a5f41eff6736274308f924d9ad8f059e6c44
Story: #2002598
Task: #22210
Depends-On: https://review.openstack.org/634646 

Upgrading to a more recent tinycore distribution to prepare
support to deliver a tinyipa image with ironic-python-agent
built with python 3
Most important changes in Tinycore 8.x include:
* kernel updated to 4.8.17
* glibc updated to 2.24
* support for Python 3.6
This patch also removes the need of get-pip.py script in
favor of native pip installation
Change-Id: I9f2f5d9faca7cbf736547b0d0ec39c5c2bb554bc
Story: #2002598
Task: #29131 

Commit 041c1795db has an issue that
prevents building IPA when requirements are not available locally.
Change-Id: I93d5fdb599315922f35a3caaf311b14d8a74a858
Story: #2004776
Task: #28901 

The new zuul jobs use just /opt/stack instead of /opt/stack/new,
breaking the constraints handling and using dependencies from source.
This changes corrects it by accepting both directories.
Change-Id: I8d6f8c99e1ab5f26e3537630d0f6086e02a2b1ec

Change-Id: I963c32407c109c840d59f4dcf945f2fc78e096f2

To fix for grenade slow discovery of e1000 network devices.
Ironic Inspector grenade job is using e1000 driver since
change: I4ec297281380896ffe9004c4de7117586ded7149
In some cases the e1000 devices has not registered yet,
and thus the dhcp init script does not find any interfaces
to bring up. This causes the grenade job to continue with
no networking, and the job finally fail.
Make the udev settle timout configurable (default 20 sec)
in finalise-tinyipa.sh.
Let's hope 20 seconds is enough. If not we now have more
logs to identify the issue.
Change-Id: I82beecc78afb4b23234d2d5eca4f46f84f20525e

This change simplifies development of IPA by allowing to install
ironib-lib from a local checkout with modifications.
Change-Id: I142c6d389d94141db8bc9e3eeaae50a687bcd7b1

Change-Id: I36b3c8c7b0dfca41db54490306f47ddbf8eec836

Adds dependency upon smartmontools's binary smartctl to
query the block devices via ATA mode which fails on pass-thru
buses such as ATA over SCSI and ATA over USB, in an effort
to prevent the initiation of ATA secure erase with one
of these interfaces in place which may render the disk
unreachable after security options are enabled for
ATA Secure Erase or upon the Secure Erase command being
sent to the Hard Disk.
Change-Id: I7635a197eb000650e919fac386b38ac15ef17041
Story: #2002546
Task: #22109
Depends-On: Ibbfd168844524d91927bdd6e67d973e0bd519bf2

Currently CI fails on build ipa image on xenial, because downloading
with wget from sourceforce is giving SSL error. Start using the
github mirror, that is not having this problem.
Additionally added autoconf, autogen, automake, and libtool
to the build requirements as the new ipmitool package is based
off of repository commit history tags and did not contain
a pregenerated configure file.
Change-Id: I139679db835b7a87c55f69a4d807ff8ec9099e3a

It seems the udhcpc script is not executable and no sleeping
cause tinyipa fails to acquire IP in multi-tenant env.
Story: #2002024
Change-Id: I3a693d75bfa54fe905bd3cd0587bb139934c087c

Increases the amount of ram for CoreOS IPA to 2GB
as the base CoreOS image is now 310MB.
Bumped CPU count for CoreOS runs to 2 CPUs as the
concurrency helps boot times for the CoreOS ramdisk.
Adds netbase, udev, and open-iscsi to debian jessie container
as they are no longer present in the default container.
Explicitly set path variable for execution in the debian
container as udevadm is in /sbin, and we may not have
/sbin on the path that is passed through to the
chroot.
Also fixed new pep8 test failures.
Story: #1600228
Task: #16287
Change-Id: I488445dfd261b7bca322a0be7b4d8ca6105750a3

Currently the generic hardware manager uses dmidecode to get the
total physical memory and system details. This patch switches the
generic hardware manager to use lshw, as it is capable of reading
more than DMI [0]. This enables systems that do not support DMI
to use the generic hardware manager, such as IBM Power systems.
[0] https://github.com/lyonel/lshw/blob/master/README.md
Closes-Bug: #1715790
Change-Id: Ie370331df6bb5ef131c5cb60f458877e2a7ad71a
Depends-On: Idaf05b8efce28cd0cbf339cf693db4f55a693d9b

The patch adds the support for rescue mode with DHCP network in
TinyIPA.
Change-Id: I10cdb47eb3815db097bb3088d9dd4804b9d6a5d0
Depends-On: I9b4c1278dc5fab7888fbfe586c15e31ed3958978
Partial-Bug: #1526449 

This is a followup to Ib6fcd5bbd7e79b9df83b660441dd685b1b17be62
It reuses more common code when adding authorized SSH keys to the
tinyipa image during rebuild and adds its directory to the list
to be cleaned.
Change-Id: Id47add60d7ad8f4266dfd2b1a3b00e45607abd5b

This is needed to produce a more unified ramdisk that can be
used for ansible-deploy with minimal rebuild
changes include:
- deprecate ENABLE_SSH var, split it to INSTALL_SSH and AUTHORIZE_SSH
 - INSTALL_SSH (defaults to true) installs and configures OpenSSH
 server
 - AUTHORIZE_SSH (defaults to false) adds SSH_PUBLIC_KEY as authorized
 for 'tc' user and makes some other changes
- symlink all from '/usr/local/(s)bin' to '/usr/(s)bin'
- adds a new script 'add-ssh-tinyipa' and 'addssh' make target
 - this effectively does those parts that finalise-tinyipa.sh script
 would do when AUTHORIZE_SSH is enabled
 - this make target will be used in gate jobs for ansible deploy
 interface
Change-Id: Ib6fcd5bbd7e79b9df83b660441dd685b1b17be62
Related-Bug: #1526308 

This patch adds support for rescue mode with DHCP tenant networks in
CoreOS. Applying network config from a configdrive is not yet supported
but will be in a future patch.
Co-Authored-By: Jay Faulkner <jay@jvf.cc>
Co-Authored-By: Taku Izumi <izumi.taku@jp.fujitsu.com>
Co-Authored-By: Annie Lezil <annie.lezil@gmail.com>
Co-Authored-By: Aparna <aparnavtce@gmail.com>
Co-Authored-By: Shivanand Tendulker <stendulker@gmail.com>
Change-Id: I7898ff22800dedba73d7fbfb3801378867abe183
Partial-Bug: 1526449

With this patch branch is added to tiny-instance image name.
Also fixed a problem with BRANCH_PATH was set to refs/heads/master
after migration to zuulv3
Change-Id: I490e2d51a7c3fbc2cc0fa78a95b984874ab934f7

To be able to test partitioned images with local boot on CI we need to
have grub installed inside user image. Unfortunately cirros doesn't
include grub-install binaries.
This patch adds ability to build partitioned image based on tinycore linux.
Related-Bug: 1657077
Change-Id: I38dce876ec9b03c0ca377f1e4d28269da8dbe316

Now that ironic-lib builds IPA from source, the IPA building procedure
needs to support requirements referring to /opt/stack/new.
As upper-constraints are not actually needed in the finalise step, they're
no longer used there to avoid dealing with file:/// references. We also
allow pre-release packages to be installed by pip.
Also install git as it's a build requirement for PBR.
Only tinyipa is updated, as it's used in the ironic-lib gate.
Change-Id: Ia942cb710eb3fe1ba56c68f91fa893795c9cb651
Closes-Bug: #1695877 

This fixes the path for ldlinux.c32. It should be in the absolute
path /usr/..., not the relative path usr/...
Related-Bug: #1694394
This is a followup to ead3b90655
Change-Id: I0aaae7e337b1ac631f789c11b3bf6e944e395456

This patch copies ldlinux.c32 to isolinux directory when building an
ISO image. ldlinux.c32 is required for syslinux 5.00 or later[1]. The
location of ldlinux.c32 depends on distrubution[2][3].
[1] http://www.syslinux.org/wiki/index.php?title=ISOLINUX#How_Can_I_Make_a_Bootable_CD_With_ISOLINUX.3F
[2] https://packages.ubuntu.com/xenial/all/syslinux-common/filelist
[3] https://www.rpmfind.net/linux/RPM/dag/redhat/el6/x86_64/extras/syslinux-5.10-1.el6.rfx.x86_64.html
Change-Id: I31dfb8423a1fe453740119ebe8b855dc23a4c033
Closes-Bug: #1694394 

This patch adds a new location of isolinux.bin for creating iso. This
file is installed to /usr/lib/ISOLINUX/ in Ubuntu 16.04 or later.
Change-Id: Iae325e3e706859b1df409f23e679d5219dc6f187
Closes-Bug: #1693127 

IPMItool is required for discovering the IPMI/BMC address during
inspection, currently its not built into TinyIPA so this patch adds the
tool to make that feature work.
Change-Id: I35bf7db3a870bc3981992e4e86d851d3f2512fd6

Adds an extra field ``biosdevname`` to network interface inventory
collected by ``default`` inspection collector (which collects the whole
inventory returned by hardware manager) of ironic-python-agent.
This feature requires biosdevname utility to collect the bios given NIC
names. The tooling module for tinyIPA is created for the same purpose.
For CoreOS IPA pxe images, biosdevname tooling module is limited,
because Docker repository is created and embedded into CoreOS pxe
images. The Docker repository uses debian to download the packages.
Debian does not have biosdevname package.
Adds an export variable TINYIPA_REQUIRE_BIOSDEVNAME. Set this
variable to ``true`` in your shell before building tinyIPA.
Closes-Bug: #1635351
Change-Id: Ia96af59e2a74868cac59e5a88cfbb3be60d85687

Commit 3c2772650e ("imagebuild: tinyipa: Build sha256 checksum files")
extended the build process to generate sha256 checksum files for all
the tinyIPA files that are uploaded to mirrors. CoreOS builds are also
uploaded to mirrors so we need to build similar sha256 checksum files
as well.
Change-Id: I5de389f14001b6957e4b0aca772edf0e93b35af8

Build sha256 checksum files which will be uploaded by the publish jobs
to https://tarballs.openstack.org/ironic-python-agent/tinyipa/files/
to provide a way to verify the integrity of the tinyipa builds.
Change-Id: I25d9cb7b83c7f04141a1b19d08585a720ae08987

Add zypper support in the install-deps.sh script so tinyipa can be
built on SUSE distributions.
Change-Id: Ie7ac36a9cab3dba825357ccd1e58290873335a68

Currently, during ramdisk optimization steps, the find tool is used to
look for files to remove. We pipe all files found by find to the rm
command running as sudo. This is necessary because of the ownership of
the files to be removed.
This commit uses the "--no-run-if-empty" GNU extension to xargs. This
extension should be present on all machines running the TinyIPA build,
because non-GNU systems simply cannot build TinyIPA.
An alternative would have been to use the -delete flag to find, and to
run find as root.
Change-Id: I78854c8901433b7d45e014dd18e7a5ff794a9c45
Closes-bug: #1661219 

Add a comment on when the TinyCore mirror list was generated to make
it more obvious when it might need to be regenerated.
Change-Id: Ie3984b5c6b85c878212497f53eb3d3eabb352523