Ephemeral disk on xenial is not mounted at boot

Bug Description

=== Begin SRU Template ===
[Impact]
An ephemeral disk will not correctly be mounted on /mnt.
This affects Azure and other instances where an ephemeral device is
mounted by default on /mnt. It is recreated in Azure and on OpenStack.

This can be mitigated by either:
1.) mount -a
2.) reboot

[Test Case]
In a correctly functioning image on OpenStack or Azure, you should be
able to:

1.) Launch an instance
2.) ssh into instance and look around
$ awk '2ドル == "/mnt" { print 0ドル }' /etc/fstab
/dev/vdb /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
$ awk '2ドル == "/mnt" { print 0ドル }' /proc/mounts
/dev/vdb /mnt vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0

$ df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/vdb 40G 32K 40G 1% /mnt

To check that this is functional with -proposed, after you've seen it failed

3.) enable -proposed and apt-get install cloud-init
4.) clean up and reboot as if fresh:
sudo rm -Rf /var/lib/cloud /var/log/cloud-init
sudo sed -i '/cloudconfig/d' /etc/fstab
sudo reboot

[Regression Potential]
This is a regression caused by bug 1611074, so in addition to the above
test case, we should go through the test cases shown there to see that
those also work.
=== End SRU Template ===

When I boot the latest xenial Azure image (containing cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.04.1), the ephemeral disk does not end up mounted (though it is formatted appropriately). Restarting the mnt.mount service does mount it, which suggests there is an issue in the ordering of the services at boot.

$ mount | grep mnt
$ sudo systemctl status mnt.mount
●くろまる mnt.mount - /mnt
Loaded: loaded (/etc/fstab; bad; vendor preset: enabled)
Active: inactive (dead)
Where: /mnt
What: /dev/disk/cloud/azure_resource-part1
Docs: man:fstab(5)
man:systemd-fstab-generator(8)
$ sudo journalctl -u mnt.mount
-- No entries --
$ cat /etc/fstab
# CLOUD_IMG: This file was created/modified by the Cloud Image build process
UUID=6a8554fa-8e1d-4916-ba03-4ca3837feb34 / ext4 defaults,discard 0 0
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2
$ sudo systemctl restart mnt.mount
$ mount | grep mnt
/dev/sdb1 on /mnt type ext4 (rw,relatime,data=ordered)
$ ls /mnt/
lost+found
$ sudo systemctl status mnt.mount
●くろまる mnt.mount - /mnt
Loaded: loaded (/etc/fstab; bad; vendor preset: enabled)
Active: active (mounted) since Tue 2016年12月06日 12:49:06 UTC; 6s ago
Where: /mnt
What: /dev/sdb1
Docs: man:fstab(5)
man:systemd-fstab-generator(8)
Process: 1916 ExecMount=/bin/mount /dev/disk/cloud/azure_resource-part1 /mnt -o defaults,x-systemd.requires=cloud-init.service,comment=cloudconfig (code=exited, status=0/SUCCESS)
Tasks: 0
Memory: 88.0K
CPU: 15ms

Dec 06 12:49:06 xenial-161206-1345 systemd[1]: Mounting /mnt...
Dec 06 12:49:06 xenial-161206-1345 systemd[1]: Mounted /mnt.

Related bugs:
* bug 1611074: Reformatting of ephemeral drive fails on resize of Azure VM
* bug 1642383: Unable to configure swap space on ephemeral disk in Azure

Status changed to 'Confirmed' because the bug affects multiple users.

seems like maybe exchanging 'mount -a' for daemon-reload might've triggered this:

https://git.launchpad.net/cloud-init/commit/cloudinit/config/cc_mounts.py?id=9e904bbc3336b96475bfd00fb3bf1262ae4de49f

This bug was fixed in the package cloud-init - 0.7.8-67-gc9c9197-0ubuntu1

---------------
cloud-init (0.7.8-67-gc9c9197-0ubuntu1) zesty; urgency=medium

* debian/cherry-pick: add utility for cherry picking commits from upstream
into patches in debian/patches.
* New upstream snapshot.
- mounts: use mount -a again to accomplish mounts (LP: #1647708)
- CloudSigma: Fix bug where datasource was not loaded in local search. (LP: #1648380)
- when adding a user, strip whitespace from group list (LP: #1354694)
- fix decoding of utf-8 chars in yaml test
- Replace usage of sys_netdev_info with read_sys_net (LP: #1625766)
- fix problems found in python2.6 test.

-- Scott Moser <email address hidden> 2016年12月11日 21:22:57 -0500

The SRU to Xenial here is blocked by 0.7.8-49-g9e904bb-0ubuntu1~16.04.2 that is currently in xenial-proposed.

Can't we merge the two uploads so as not to delay things until one or the other is out of proposed?

Discussion on IRC was to not disrupt the current testing of xenial-proposed. Robie has already reviewed this SRU and will process it as soon as 0.7.8-49-g9e904bb-0ubuntu1~16.04.2 clears (should be Monday assuming all verification/testing is done.

Hello Dan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-49-g9e904bb-0ubuntu1~16.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I've verified this as described in the test case.

## fresh instance
$ dpkg-query --show cloud-init
0.7.8-49-g9e904bb-0ubuntu1~16.04.1

$ cat /etc/cloud/build.info
build_name: server
serial: 20161213

## /proc/mountsn does not have the /mnt mount, but /etc/fstab does
$ awk '2ドル == "/mnt" { print 0ドル }' /proc/mounts
$ awk '2ドル == "/mnt" { print 0ドル }' /etc/fstab
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

$ m=http://azure.archive.ubuntu.com/ubuntu/
$ echo deb $m xenial-proposed main universe |
sudo tee /etc/apt/sources.list.d/proposed.list

$ sudo apt-get update && sudo apt-get install -qy cloud-init

$ dpkg-query --show cloud-init
cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.04.3

## Clean up for new reboot
# remove old entry in /etc/fstab
$ sudo sudo sed -i '/cloudconfig/d' /etc/fstab
$ sudo umount /mnt

## reformat to ntfs to look like first boot
$ disk=/dev/disk/cloud/azure_resource
$ echo "2048,1024000,7" | sudo sfdisk --force "$disk"
$ sudo udevadm settle
$ sudo umount /mnt
$ time sudo mkfs.ntfs --quick ${disk}-part1

## remove state
$ sudo rm -Rf /var/lib/cloud/ /var/log/cloud-init*
$ sudo reboot

## ssh back in ..
$ awk '2ドル == "/mnt" { print 0ドル }' /proc/mounts
/dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0
$ awk '2ドル == "/mnt" { print 0ドル }' /etc/fstab
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

$ grep Activ /var/log/cloud-init.log
2016年12月20日 18:52:41,270 - cc_mounts.py[DEBUG]: Activate mounts: PASS:mount -a
2016年12月20日 18:52:41,507 - cc_mounts.py[DEBUG]: Activate mounts: PASS:systemctl daemon-reload

## for good measure, reboot again (to similate non-first boot).
$ sudo reboot

## ssh back in, look at /mnt is mounted
$ awk '2ドル == "/mnt" { print 0ドル }' /proc/mounts
/dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0

I've validated that after installing the cloud-init from -proposed, and rebooting that the ephemeral disk was mounted at /mnt.

(and this validation was done on the latest daily xenial image)

This bug was fixed in the package cloud-init - 0.7.8-49-g9e904bb-0ubuntu1~16.04.3

---------------
cloud-init (0.7.8-49-g9e904bb-0ubuntu1~16.04.3) xenial-proposed; urgency=medium

* debian/cherry-pick: use git format-patch rather than git show
* cherry-pick a9d41de: CloudSigma: Fix bug where datasource was not
loaded in local (LP: #1648380)
* cherry-pick c9c9197: mounts: use mount -a again to accomplish mounts
(LP: #1647708)

-- Scott Moser <email address hidden> 2016年12月13日 16:02:50 -0500

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This is fixed in cloud-init 0.7.9.

Hello Dan, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-68-gca3ae67-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

## fresh instance
$ dpkg-query --show cloud-init
cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.10.1

$ cat /etc/cloud/build.info
build_name: server
serial: 20170114

## /proc/mounts does not have the /mnt mount, but /etc/fstab does
$ awk '2ドル == "/mnt" { print 0ドル }' /proc/mounts
$ awk '2ドル == "/mnt" { print 0ドル }' /etc/fstab
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

## enable proposed update
$ m=http://azure.archive.ubuntu.com/ubuntu/
$ echo deb $m $(lsb_release -sc)-proposed main universe |
sudo tee /etc/apt/sources.list.d/proposed.list

$ sudo apt-get update && sudo apt-get install -qy cloud-init

$ dpkg-query --show cloud-init
cloud-init 0.7.8-68-gca3ae67-0ubuntu1~16.10.1

## Clean up for new reboot
# remove old entry in /etc/fstab
$ sudo sudo sed -i '/cloudconfig/d' /etc/fstab
$ sudo umount /mnt
umount: /mnt: not mounted

## reformat to ntfs to look like first boot
$ disk=/dev/disk/cloud/azure_resource
$ echo "2048,1024000,7" | sudo sfdisk --force "$disk"
$ sudo udevadm settle
$ sudo umount /mnt
$ blockdev --rereadpt "$disk"
$ time sudo mkfs.ntfs --quick ${disk}-part1

## remove state
$ sudo rm -Rf /var/lib/cloud/ /var/log/cloud-init*
$ sudo reboot

## ssh back in
$ awk '2ドル == "/mnt" { print 0ドル }' /proc/mounts
/dev/sdb1 /mnt ext4 rw,relatime,stripe=4,data=ordered 0 0
$ awk '2ドル == "/mnt" { print 0ドル }' /etc/fstab
/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2

$ grep Activ /var/log/cloud-init.log
2017年01月20日 17:16:51,201 - cc_mounts.py[DEBUG]: Activate mounts: PASS:mount -a
2017年01月20日 17:16:51,262 - cc_mounts.py[DEBUG]: Activate mounts: PASS:systemctl daemon-reload

# for good measure
$ sudo reboot

## ssh back in
$ awk '2ドル == "/mnt" { print 0ドル }' /proc/mounts
/dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0

This bug was fixed in the package cloud-init - 0.7.8-68-gca3ae67-0ubuntu1~16.10.1

---------------
cloud-init (0.7.8-68-gca3ae67-0ubuntu1~16.10.1) yakkety; urgency=medium

* debian/cherry-pick: add utility for cherry picking commits from upstream
into patches in debian/patches.
* New upstream snapshot.
- mounts: use mount -a again to accomplish mounts (LP: #1647708)
- CloudSigma: Fix bug where datasource was not loaded in local search.
(LP: #1648380)
- when adding a user, strip whitespace from group list
[Lars Kellogg-Stedman] (LP: #1354694)
- fix decoding of utf-8 chars in yaml test
- Replace usage of sys_netdev_info with read_sys_net (LP: #1625766)
- fix problems found in python2.6 test.
- OpenStack: extend physical types to include hyperv, hw_veb, vhost_user.
(LP: #1642679)
- tests: fix assumptions that expected no eth0 in system. (LP: #1644043)
- net/cmdline: Consider ip= or ip6= on command line not only ip=
(LP: #1639930)
- Just use file logging by default [Joshua Harlow] (LP: #1643990)
- Improve formatting for ProcessExecutionError [Wesley Wiedenmeier]
- flake8: fix trailing white space
- Doc: various documentation fixes [Sean Bright]
- cloudinit/config/cc_rh_subscription.py: Remove repos before adding
[Brent Baude]
- packages/redhat: fix rpm spec file.
- main: set TZ in environment if not already set. [Ryan Harper]
- disk_setup: Use sectors as unit when formatting MBR disks with sfdisk.
[Daniel Watkins] (LP: #1460715)

-- Scott Moser <email address hidden> 2016年12月19日 15:07:12 -0500

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2781