growing root partition does not always work with root=PARTUUID=

Bug Description

=== Begin SRU Template ===
[Impact]
Growing the root partition would fail in either of two cases:
a.) if the device /dev/root existed
This occurs on yakkety systems.

b.) the kernel command line had upper case letters in PARTUUID=<value>
This can be fed via root= on either xenial or yakkety.

[Test Case]
get-proposed-image is
https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/tree/bin/get-proposed-image
It downloads a cloud image of a given release, and then creates a -proposed
image with cloud-init upgraded.

1.) get a (proposed) disk image image.
and convert it to raw so you can read the partuuid with sfdisk
(get-proposed-image does this, if not,
'qemu-img convert -O raw orig.img orig.raw')
./get-proposed-image

2.) get the partition uuid of the first partition
$ raw=yakkety-server-cloudimg-amd64-proposed.raw
$ ptuuid=$(sfdisk --part-uuid $raw 1)
# sfdisk will normally output in upper case, but be explicit here.
$ ptuuid=$(echo "$ptuuid" | tr '[a-z]' '[A-Z]')

3.) create a nocloud seed
$ printf "%s\n%s\n%s\n%s\n" "#cloud-config" "password: passw0rd" \
"chpasswd: {expire: False}" "ssh_pwauth: True" > my-user-data
$ echo "instance-id: $(uuidgen || echo i-abcdefg)" > my-meta-data
$ cloud-localds my-seed.img my-user-data my-meta-data

4.) extract kernel from inside the image
$ sudo mount-image-callback $raw -- mchroot sh -xc 'cat /boot/vmlinu?-*' > kernel

5.) boot instance with disk backed by the raw disk above.

$ qemu-img create -f qcow2 -b $raw disk.img 10G
$ qemu-system-x86_64 -enable-kvm \
-drive file=disk.img,if=ide,index=0 -drive file=my-seed.img,if=ide \
-net nic -net user,hostfwd=tcp::2222-:22 \
-snapshot -m 768 -nographic -echr 0x05 \
-kernel kernel \
-append "root=PARTUUID=${ptuuid} ro console=tty1 console=ttyS0"

6.) log in, verify / has been resized.
log in with 'ubuntu' and password 'passw0rd'
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/root 9.6G 1009M 8.6G 11% /

[Regression Potential]
The regression path is really the case where devent2dev finds /dev/root
and /dev/root exists. In that case, we now possibly return a /dev/<device>
path when previously it would have returned /dev/root.

[Other Info]
The qemu-system-x86 command above uses ide devices. This is because
the ide device emulated by qemu is built into the -generic kernel,
whilei the more common virtio-block or virtio-scsi are not. If you
attach those device types, it will fail with 'cant find root'.

=== End SRU Template ===

When trying to verify I found a couple cases where this still does not work.

Related bugs:
* bug 1677376: growing partitions does not work when booted without initramfs
* bug 1685291: RFC: virtio and virtio-scsi should be built in
* bug 1725067: cloud-init resizefs fails when booting with root=PARTUUID= Edit

This bug was fixed in the package cloud-init - 0.7.9-113-g513e99e0-0ubuntu1

---------------
cloud-init (0.7.9-113-g513e99e0-0ubuntu1) artful; urgency=medium

* New upstream snapshot.
- nova-lxd: read product_name from environment, not platform.
(LP: #1685810)
- Fix yum repo config where keys contain array values
[Dylan Perry] (LP: #1592150)
- template: Update debian backports template [Joshua Powers] (LP: #1627293)
- rsyslog: replace ~ with stop [Joshua Powers] (LP: #1367899)
- Doc: add additional RTD examples [Joshua Powers] (LP: #1459604)
- Fix growpart for some cases when booted with root=PARTUUID.
(LP: #1684869)
- pylint: update output style to parseable [Joshua Powers]
- pylint: fix all logging warnings [Joshua Powers]
- CloudStack: Add NetworkManager to list of supported DHCP lease dirs.
[Syed Mushtaq Ahmed]
- net: kernel lies about vlans not stealing mac addresses, when they do
[Dimitri John Ledkov] (LP: #1682871)
- ds-identify: Check correct path for "latest" config drive
[Daniel Watkins] (LP: #1673637)
- doc: Fix example for resolve.conf configuration.
[Jon Grimm] (LP: #1531582)
- Fix examples that reference upstream chef repository.
[Jon Grimm] (LP: #1678145)
- doc: correct grammar and improve clarity in merging documentation.
[David Tagatac]
- doc: Add missing doc link to snap-config module. [Ryan Harper]
- snap: allows for creating cloud-init snap [Joshua Powers]
- DigitalOcean: assign IPv4ll address to lowest indexed interface.
[Ben Howard]
- DigitalOcean: configure all NICs presented in meta-data. [Ben Howard]
- Remove (and/or fix) URL shortener references [Jon Grimm] (LP: #1669727)
- HACKING.rst: more info on filling out contributors agreement.
[Scott Moser]
- util: teach write_file about copy_mode option
[Lars Kellogg-Stedman] (LP: #1644064)
- DigitalOcean: bind resolvers to loopback interface. [Ben Howard]
- tests: fix AltCloud tests to not rely on blkid (LP: #1636531)

-- Scott Moser <email address hidden> 2017年4月25日 16:34:08 -0400

I don't think this is a correct test case for xenial; since the standard kvm images use the release kernel, which is 4.4, rather than the hwe-rolling kernel (4.8), the behavior will match the other 4.4-based image I already tested. We need a test case that will actually fail with the current cloud-init and succeed with the new one, which AIUI requires an image using the rolling kernel in xenial.

The test case now explicitly uses upper case uuids, so we are testing that part of the fix on xenial. The /dev/root fix will be tested for yakkety only (though it will also ultimately benefit xenial users w/ the rolling kernel).

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-113-g513e99e0-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Scott, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-113-g513e99e0-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Scott, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-113-g513e99e0-0ubuntu1~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Validated yakkety and zesty:
$ raw=yakkety-server-cloudimg-amd64.raw
$ yakkety-server-cloudimg-amd64.raw
yakkety-server-cloudimg-amd64.raw: command not found
$ sfdisk --part-uuid $raw 1
F9E1FDF2-234A-4732-AEF8-A75B86313FAD
$ ptuuid=$(sfdisk --part-uuid $raw 1)
$ ptuuid=$(echo "$ptuuid" | tr '[a-z]' '[A-Z]')
$ echo $ptuuid
F9E1FDF2-234A-4732-AEF8-A75B86313FAD
$ printf "%s\n%s\n%s\n%s\n" "#cloud-config" "password: passw0rd" \
> "chpasswd: {expire: False}" "ssh_pwauth: True" > my-user-data
$ echo "instance-id: $(uuidgen || echo i-abcdefg)" > my-meta-datacsmith@fringe:~$ cloud-localds my-seed.img my-user-data my-meta-data
$ sudo mount-image-callback $raw -- mchroot sh -xc 'cat /boot/vmlinu?-*' > kernel
[sudo] password for csmith:
+ cat /boot/vmlinuz-4.8.0-51-generic
$ qemu-img create -f qcow2 -b $raw disk.img 10G
Formatting 'disk.img', fmt=qcow2 size=10737418240 backing_file=yakkety-server-cloudimg-amd64.raw encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/root 2.0G 1011M 984M 51% /

$ dpkg -l cloud-init
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii cloud-init 0.7.9-90-g61 all Init scripts for cloud instances
ubuntu@ubuntu:~$ grep VERSION= /etc/os-release
VERSION="16.10 (Yakkety Yak)"

$ raw=zesty-server-cloudimg-amd64.raw
$ sfdisk --part-uuid $raw 149750BCB-3B61-4F13-92D4-D3153E9C28C6
$ ptuuid=$(sfdisk --part-uuid $raw 1)csmith@fringe:~$ ptuuid=$(echo "$ptuuid" | tr '[a-z]' '[A-Z]')
$ printf "%s\n%s\n%s\n%s\n" "#cloud-config" "password: passw0rd" \
> "chpasswd: {expire: False}" "ssh_pwauth: True" > my-user-data
$ echo "instance-id: $(uuidgen || echo i-abcdefg)" > my-meta-datacsmith@fringe:~$ cloud-localds my-seed.img my-user-data my-meta-data
$ sudo mount-image-callback $raw -- mchroot sh -xc 'cat /boot/vmlinu?-*' > kernel
[sudo] password for csmith:
+ cat /boot/vmlinuz-4.10.0-20-generic
$ qemu-img create -f qcow2 -b $raw disk.img 10G
Formatting 'disk.img', fmt=qcow2 size=10737418240 backing_file=zesty-server-cloudimg-amd64.raw encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-system-x86_64 -enable-kvm \
> -drive file=disk.img,if=ide,index=0 -drive file=my-seed.img,if=ide \
> -net nic -net user,hostfwd=tcp::2222-:22 \
> -snapshot -m 768 -nographic -echr 0x05 \
> -kernel kernel \
> -append "root=PARTUUID=${ptuuid} ro console=tty1 console=ttyS0"
# login ubuntu/passw0rd

ubuntu@ubuntu:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/root 2.0G 988M 1007M 50% /

for xenial you need to download the '-uefi1.img' in order to get
a disk that is partitioned with a GPT table.

Given a small modification to get-proposed-cloudimg:

--- get-proposed-cloudimg.dist 2017年05月16日 13:56:05.000000000 +0000
+++ get-proposed-cloudimg 2017年05月16日 13:52:49.192688492 +0000
@@ -6,7 +6,8 @@
fname=$rel-server-cloudimg-amd64.img
ofname="$fname"
case "$rel" in
- precise|trusty|xenial) ofname="$rel-server-cloudimg-amd64-disk1.img"
+ precise|trusty) ofname="$rel-server-cloudimg-amd64-disk1.img";;
+ xenial) ofname="$rel-server-cloudimg-amd64-uefi1.img"
esac
pfname="${fname%.img}-proposed.img"
raw="${fname%.img}.raw"

Then, I just followed along with the script.

$ ls -l xenial-server-*
-rw-rw-r-- 1 smoser smoser 334823424 May 15 15:32 xenial-server-cloudimg-amd64.img
-rw-r--r-- 1 smoser smoser 102825984 May 16 13:53 xenial-server-cloudimg-amd64-proposed.img
-rw-r--r-- 1 smoser smoser 2361393152 May 16 13:53 xenial-server-cloudimg-amd64.raw

$ raw=xenial-server-cloudimg-amd64.raw
$ sfdisk --part-uuid $raw 1
7D8DBC14-97B9-4F0F-ADD0-F1E22F1DE4AD
$ ptuuid=$(sfdisk --part-uuid $raw 1)
$ ptuuid=$(echo "$ptuuid" | tr '[a-z]' '[A-Z]')
$ echo $ptuuid
7D8DBC14-97B9-4F0F-ADD0-F1E22F1DE4AD

$ printf "%s\n%s\n%s\n%s\n" "#cloud-config" "password: passw0rd" \
"chpasswd: {expire: False}" "ssh_pwauth: True" > my-user-data
$ echo "instance-id: $(uuidgen || echo i-abcdefg)" > my-meta-data

$ sudo mount-image-callback $raw -- mchroot sh -xc 'cat /boot/vmlinu?-*' > kernel
+ cat /boot/vmlinuz-4.4.0-77-generic

$ qemu-img create -f qcow2 -b xenial-server-cloudimg-amd64-proposed.img disk.img 10G
Formatting 'disk.img', fmt=qcow2 size=10737418240 backing_file=xenial-server-cloudimg-amd64-proposed.img encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

$ qemu-system-x86_64 -enable-kvm \
-drive file=disk.img,if=ide,index=0 -drive file=my-seed.img,if=ide \
-net nic -net user,hostfwd=tcp::2222-:22 \
-snapshot -m 768 -nographic -echr 0x05 -snapshot \
-kernel kernel \
-append "root=PARTUUID=${ptuuid} ro console=tty1 console=ttyS0"

... <see attachment> ...

ubuntu@ubuntu:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/root 9.6G 963M 8.6G 10% /
ubuntu@ubuntu:~$ dpkg-query --show cloud-init
cloud-init 0.7.9-113-g513e99e0-0ubuntu1~16.04.1
ubuntu@ubuntu:~$ lsb_release -sc
xenial
ubuntu@ubuntu:~$ uname -r
4.4.0-77-generic
ubuntu@ubuntu:~$ cat /proc/cmdline
root=PARTUUID=7D8DBC14-97B9-4F0F-ADD0-F1E22F1DE4AD ro console=tty1 console=ttyS0

This bug was fixed in the package cloud-init - 0.7.9-113-g513e99e0-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-113-g513e99e0-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

* debian/update-grub-legacy-ec2: fix early exit failure no /etc/fstab
file. (LP: #1682160)
* New upstream snapshot.
- nova-lxd: read product_name from environment, not platform.
(LP: #1685810)
- Fix yum repo config where keys contain array values [Dylan Perry]
- template: Update debian backports template [Joshua Powers]
- rsyslog: replace ~ with stop [Joshua Powers] (LP: #1367899)
- Doc: add additional RTD examples [Joshua Powers]
- Fix growpart for some cases when booted with root=PARTUUID.
(LP: #1684869)
- pylint: update output style to parseable [Joshua Powers]
- pylint: fix all logging warnings [Joshua Powers]
- CloudStack: Add NetworkManager to list of supported DHCP lease dirs.
[Syed Mushtaq Ahmed]
- net: kernel lies about vlans not stealing mac addresses, when they do
[Dimitri John Ledkov] (LP: #1682871)
- ds-identify: Check correct path for "latest" config drive
[Daniel Watkins] (LP: #1673637)
- doc: Fix example for resolv.conf configuration. [Jon Grimm]
- Fix examples that reference upstream chef repository. [Jon Grimm]
- doc: correct grammar and improve clarity in merging documentation.
[David Tagatac]
- doc: Add missing doc link to snap-config module. [Ryan Harper]
- snap: allows for creating cloud-init snap [Joshua Powers]
- DigitalOcean: assign IPv4ll address to lowest indexed interface.
[Ben Howard] (LP: #1676908)
- DigitalOcean: configure all NICs presented in meta-data.
[Ben Howard] (LP: #1676908)
- Remove (and/or fix) URL shortener references [Jon Grimm]
- HACKING.rst: more info on filling out contributors agreement.
- util: teach write_file about copy_mode option
[Lars Kellogg-Stedman] (LP: #1644064)
- DigitalOcean: bind resolvers to loopback interface.
[Ben Howard] (LP: #1676908)
- tests: fix AltCloud tests to not rely on blkid (LP: #1636531)

-- Scott Moser <email address hidden> 2017年4月27日 12:51:04 -0400

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package cloud-init - 0.7.9-113-g513e99e0-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-113-g513e99e0-0ubuntu1~16.10.1) yakkety; urgency=medium

* debian/update-grub-legacy-ec2: fix early exit failure no /etc/fstab
file. (LP: #1682160)
* New upstream snapshot.
- nova-lxd: read product_name from environment, not platform.
(LP: #1685810)
- Fix yum repo config where keys contain array values [Dylan Perry]
- template: Update debian backports template [Joshua Powers]
- rsyslog: replace ~ with stop [Joshua Powers] (LP: #1367899)
- Doc: add additional RTD examples [Joshua Powers]
- Fix growpart for some cases when booted with root=PARTUUID.
(LP: #1684869)
- pylint: update output style to parseable [Joshua Powers]
- pylint: fix all logging warnings [Joshua Powers]
- CloudStack: Add NetworkManager to list of supported DHCP lease dirs.
[Syed Mushtaq Ahmed]
- net: kernel lies about vlans not stealing mac addresses, when they do
[Dimitri John Ledkov] (LP: #1682871)
- ds-identify: Check correct path for "latest" config drive
[Daniel Watkins] (LP: #1673637)
- doc: Fix example for resolv.conf configuration. [Jon Grimm]
- Fix examples that reference upstream chef repository. [Jon Grimm]
- doc: correct grammar and improve clarity in merging documentation.
[David Tagatac]
- doc: Add missing doc link to snap-config module. [Ryan Harper]
- snap: allows for creating cloud-init snap [Joshua Powers]
- DigitalOcean: assign IPv4ll address to lowest indexed interface.
[Ben Howard] (LP: #1676908)
- DigitalOcean: configure all NICs presented in meta-data.
[Ben Howard] (LP: #1676908)
- Remove (and/or fix) URL shortener references [Jon Grimm]
- HACKING.rst: more info on filling out contributors agreement.
- util: teach write_file about copy_mode option
[Lars Kellogg-Stedman] (LP: #1644064)
- DigitalOcean: bind resolvers to loopback interface.
[Ben Howard] (LP: #1676908)
- tests: fix AltCloud tests to not rely on blkid (LP: #1636531)

-- Scott Moser <email address hidden> 2017年4月27日 13:38:40 -0400

This bug was fixed in the package cloud-init - 0.7.9-113-g513e99e0-0ubuntu1~17.04.1

---------------
cloud-init (0.7.9-113-g513e99e0-0ubuntu1~17.04.1) zesty; urgency=medium

* debian/update-grub-legacy-ec2: fix early exit failure no /etc/fstab
file. (LP: #1682160)
* New upstream snapshot.
- nova-lxd: read product_name from environment, not platform.
(LP: #1685810)
- Fix yum repo config where keys contain array values [Dylan Perry]
- template: Update debian backports template [Joshua Powers]
- rsyslog: replace ~ with stop [Joshua Powers] (LP: #1367899)
- Doc: add additional RTD examples [Joshua Powers]
- Fix growpart for some cases when booted with root=PARTUUID.
(LP: #1684869)
- pylint: update output style to parseable [Joshua Powers]
- pylint: fix all logging warnings [Joshua Powers]
- CloudStack: Add NetworkManager to list of supported DHCP lease dirs.
[Syed Mushtaq Ahmed]
- net: kernel lies about vlans not stealing mac addresses, when they do
[Dimitri John Ledkov] (LP: #1682871)
- ds-identify: Check correct path for "latest" config drive
[Daniel Watkins] (LP: #1673637)
- doc: Fix example for resolv.conf configuration. [Jon Grimm]
- Fix examples that reference upstream chef repository. [Jon Grimm]
- doc: correct grammar and improve clarity in merging documentation.
[David Tagatac]
- doc: Add missing doc link to snap-config module. [Ryan Harper]
- snap: allows for creating cloud-init snap [Joshua Powers]
- DigitalOcean: assign IPv4ll address to lowest indexed interface.
[Ben Howard] (LP: #1676908)
- DigitalOcean: configure all NICs presented in meta-data.
[Ben Howard] (LP: #1676908)
- Remove (and/or fix) URL shortener references [Jon Grimm]
- HACKING.rst: more info on filling out contributors agreement.
- util: teach write_file about copy_mode option
[Lars Kellogg-Stedman] (LP: #1644064)
- DigitalOcean: bind resolvers to loopback interface.
[Ben Howard] (LP: #1676908)
- tests: fix AltCloud tests to not rely on blkid (LP: #1636531)

-- Scott Moser <email address hidden> 2017年4月27日 15:09:31 -0400

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

I updated bug the description to mention bug 1725067 which was a regression that is shown with the recreate above.

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2863