openstack/ironic-python-agent - ironic-python-agent - OpenDev: Free Software Needs Free Tools

Now that ironic-lib builds IPA from source, the IPA building procedure
needs to support requirements referring to /opt/stack/new.
As upper-constraints are not actually needed in the finalise step, they're
no longer used there to avoid dealing with file:/// references. We also
allow pre-release packages to be installed by pip.
Also install git as it's a build requirement for PBR.
Only tinyipa is updated, as it's used in the ironic-lib gate.
Change-Id: Ia942cb710eb3fe1ba56c68f91fa893795c9cb651
Closes-Bug: #1695877 

IPMItool is required for discovering the IPMI/BMC address during
inspection, currently its not built into TinyIPA so this patch adds the
tool to make that feature work.
Change-Id: I35bf7db3a870bc3981992e4e86d851d3f2512fd6

Adds an extra field ``biosdevname`` to network interface inventory
collected by ``default`` inspection collector (which collects the whole
inventory returned by hardware manager) of ironic-python-agent.
This feature requires biosdevname utility to collect the bios given NIC
names. The tooling module for tinyIPA is created for the same purpose.
For CoreOS IPA pxe images, biosdevname tooling module is limited,
because Docker repository is created and embedded into CoreOS pxe
images. The Docker repository uses debian to download the packages.
Debian does not have biosdevname package.
Adds an export variable TINYIPA_REQUIRE_BIOSDEVNAME. Set this
variable to ``true`` in your shell before building tinyIPA.
Closes-Bug: #1635351
Change-Id: Ia96af59e2a74868cac59e5a88cfbb3be60d85687

Build sha256 checksum files which will be uploaded by the publish jobs
to https://tarballs.openstack.org/ironic-python-agent/tinyipa/files/
to provide a way to verify the integrity of the tinyipa builds.
Change-Id: I25d9cb7b83c7f04141a1b19d08585a720ae08987

Add zypper support in the install-deps.sh script so tinyipa can be
built on SUSE distributions.
Change-Id: Ie7ac36a9cab3dba825357ccd1e58290873335a68

Currently, during ramdisk optimization steps, the find tool is used to
look for files to remove. We pipe all files found by find to the rm
command running as sudo. This is necessary because of the ownership of
the files to be removed.
This commit uses the "--no-run-if-empty" GNU extension to xargs. This
extension should be present on all machines running the TinyIPA build,
because non-GNU systems simply cannot build TinyIPA.
An alternative would have been to use the -delete flag to find, and to
run find as root.
Change-Id: I78854c8901433b7d45e014dd18e7a5ff794a9c45
Closes-bug: #1661219 

Add a comment on when the TinyCore mirror list was generated to make
it more obvious when it might need to be regenerated.
Change-Id: Ie3984b5c6b85c878212497f53eb3d3eabb352523

this patch adds a script with known Tinycore Linux HTTP mirrors
and if TINYCORE_MIRROR_URL env var was not provided explicitly,
sets in to the first URL from the list that responds to HEAD request.
The HEAD request is performed without retries and with timeout of 10
seconds.
The list of mirrors is taken from official TC wiki [0].
Currently not responding mirrors are moved to the bottom of the list,
ant those that currently respond with 404 are moved to the very bottom.
The main official repo is left at the first mirror to try.
[0] http://wiki.tinycorelinux.net/wiki:mirrors
Change-Id: I539db06baf1d27b428b75d412851367bfe8a2608

Updated file tox.ini allows to check code style in all ".sh" files
included in Ironic Python Agent. Checking can be invoked by calling either
"tox" or "tox -e pep8".
Change-Id: Ie76605737c7db10a064f2aebfda388372a4c0591
Closes-Bug: #1625215 

This produces urls with `//` which works but looks strange.
Change-Id: I9ff9ec940d7f814df669c752c5f7a6d7380b8175

Somtimes it is useful to have tinyipa image with normal Python
experience (access to source code for debugging, Python interpreter).
This patch allows to skip compiling all Python to *.pyo and
removing all *.py[c] files by setting
export PYOPTIMIZE_TINYIPA=false
in the shell before running the build scripts.
Change-Id: I570d5a30af0f6a0c9de533e61d03ad5c198c2864
Closes-Bug: #1629953 

might be useful sometimes,
for example debugging in normal terminal is usually more pleasant
than in QEMU console.
Add env variables `ENABLE_SSH` and `SSH_PUBLIC_KEY`,
documentation is amended accordingly.
Change-Id: I71352c2087e1fb9a7174affb78fe695932a924d7

Change-Id: I4561116b4f8f2b60ebab9ad58ece8c2223f44abb

Currently, building ramdisks installs ironic-python-agent without any
upper-constraints. This causes the package to be installed with newer,
untested dependencies.
This commits introduces a tool to generate a local upper-constraints
file based on predefined strategies (below). Additionally, the fallback
to the openstack/requirements uses the URL defined in tox.ini instead of
redefining it. This prevents having to keep track of two separate
variables when releasing.
upper-constraints lookup strategies (in order):
 * UPPER_CONSTRAINTS_FILE points to a local file
 * UPPER_CONSTRAINTS_FILE points to a URL
 * /opt/stack/new/requirements/upper-constraints.txt
 * upper-constraints.txt from openstack/requirements git repository
Partial-bug: #1616554
Change-Id: Ib5c0c57cafdb6ffd7456e61f3b1bb5fa57520e5a

Currently image build for tinyipa overrides resolv.conf with the one
from build host instead of backing it up.
Change-Id: I18598fe5327d0ff74cac2596161b9201d47d2c5b

The log extension is responsible for retrieving logs from the system,
if journalctl is present the logs will come from it, otherwise we
fallback to getting the logs from the /var/log directory + dmesg logs.
In the coreos ramdisk, we need to bind mount /run/log in the container
so the IPA service can have access to the journal.
For the tinyIPA ramdisk, the logs from IPA are now being redirected to
/var/logs/ironic-python-agent.log instead of only going to the default
stdout.
Inspector now shares the same method of collecting logs, extending its
capabilities for non-systemd systems.
Partial-Bug: #1587143
Change-Id: Ie507e2e5c58cffa255bbfb2fa5ffb95cb98ed8c4

Add additional "git checkout" to tag to make sure
we are on the currect tag. This will
allow to support OSes with git >= v1.6.5 e.g. CentOS 6.7
Closes-bug: #1590912
Change-Id: I87a87e1148a0590fa7e378296ce00e0a28666227

Installing packages into a chroot was not something tce-load was ever
designed for, because of this it gets a little confused about loading
kernel modules when we install them. This patch manually calls depmod to
ensure that the kernel modules we install during the build process get
loaded when we boot the ramdisk.
Change-Id: I940d11f3f9f2a39cba16e121dc1809a42690d350

Change-Id: Idc8092f4cce2a567353b1e70e8f56813471a6dbb

This is a dep of gdisk, however the gdisk deps file was removed
upstream. Add this back manually for now.
Change-Id: Ib2beb8688a22873fd39653d26571ceb0fb70447d

This makes the TinyCoreLinux build mirror configurable, and makes the
default the official repository, http://repo.tinycorelinux.net/. This
mirror consistently tested 15% faster than using ibiblio.org for all
tasks.
Additionally, ensure it is used throughout the build process.
This will allow us to point to an infra-controlled mirror down the
road [1].
[1] http://lists.openstack.org/pipermail/openstack-infra/2016-May/004277.html
Change-Id: I03c98517e2e8aa08e18e059c3fe84b7cdef1a3a2

This patch aims to reduce the TinyIPA ramdisk file size by removing
some files that aren't required to run IPA once built and also striping
the binaries we build for qemu-utils and tgt to make them as small as
possible.
Change-Id: I6b8d648800284444d67a6988511707e57de1e6c9

This patch adds code into the tinyipa build process and IPA itself to
allow the required python code to be PYTHONOPTIMIZE precompiled into
pyo files, this speeds up IPA startup time in a nested virt by 50%.
Change-Id: Ib60c420719ea52a602c1752b572d3b217c2cefc7

Bump to the latest version mainly to get the slimmed down gdisk [1].
Additionally, ensure /dev/pts is mounted in the chroot to avoid make
from segfaulting when ttyname fails [2]. This only affects interactive
mode.
[1]: http://forum.tinycorelinux.net/index.php/topic,19951.0.html
[2]: http://patches.openembedded.org/patch/117921/
Change-Id: If752cd9ed561a571c9690eb7390c3ee3910b09e0

Change-Id: I81d2b52f7a5733a833fec9fcce0667f9f4bf5620

The build of tinyipa is doing a: git clone git://...
Doing a git clone git://... behind a proxy server is painful. Use the
'official' QEMU github mirror and clone with https.
Change-Id: Ia14835e6867b945cccb8b1987aa4881d8524561c
Closes-Bug: #1578778 

This patch reduces tinyipa boot time significantly by preinstalling all
required system packages at build time instead of the traditional
tinycorelinux method of loading them at runtime, cutting many seconds
of unpacking squashfs files out of the boot procedure.
Change-Id: If10cb64ea6b64972df0cdbd1056d84f136fec94a

Squashfs-tools is missing when building the tinyipa image during the
post job, this patch adds a new script to the tinyipa build process that
ensures that missing dependencies get installed as part of the build
process. It also replaces a use of realpath it build-iso.sh where it
should be readlink to ensure that we don't need that dependency.
Change-Id: I9a60dcf42f6a9ba624aa347f0f375bb40432ef77
Closes-Bug: #1567981 

For publishing tinyipa from stable/branches we need to be able to place
branch references into the files created so that when they are uploaded
versions from different branches do not conflict. This change also adds
building a tar for publishing too.
Change-Id: I9d879689c8e88768aaa96d0033b62d062f9cdc27

Binding to the first interface that has an IP address is error-prone: there is
no guarantee that ironic can reach us via this inteface. It is much safer to
detect the interface facing ironic and bind to it.
Unused LookupAgentInterfaceError exception is deleted.
The TinyIPA build also requires iptables dependency at build time to insert the
required kernel modules.
Closes-Bug: #1558956
Change-Id: I9586805e6c7f52a50834bc03efeb72d1faa6cb65

During the tinyipa build process we use the tinyipa package manager,
tce-load. When tce-load trys to look for packages that are kernel specfic
it uses uname to calculate the kernel version, and because we're
building in a chroot it detects the hosts kernel version instead of the
kernel version we are actually going to be using for the final tinyipa
image, this causes it to fail to find certain packages. This patch
injects a fake uname script into the tinyipa build and finalise
environments to simulate running under the correct kernel version.
Change-Id: I2b483806cbb1786763e32f3ccb7f88a6deeee3d1

This patch update the tinyipa script to work
on Fedora. First issue I add was that chroot --userspec=
didn't accepted the ts:staff, but work with uid and guid.
Also I added env -i to start empty environment var
for some reason my environment var mess up the build
Change-Id: I5ed1995ba8a6d30af8e3a3fb51286e689cc15df9

We want to install the dev version of IPA but we don't want pre-release
versions of IPA's dependencies, this patch ensures that only released
deps are installed.
Change-Id: I6187fbbd0bdd80503861d3760248f3f23c1c6388

We need to explicitly use /bin/bash for the {build, finalise}-tinyipa.sh
scripts because the script relies on a bash extension to get the current
working directory.
Depends-On: I4690514803dd91d4bda0d38df4fd792f31a51421
Change-Id: Ic1ed66c8fa8936a822e112199844a869106751dc

TinyCoreLinux is a small 10mb base linux distribution which can easily
be built into a ramdisk for PXE and ISO booting. This patch adds scripts
and other required files to build a TinyCore based ramdisk which runs
the Ironic Python Agent.
Change-Id: I79fbec75de19b3d6feb19f87a3a0f662eae42bf3