cloud-init

Azure reboot with unformatted ephemeral drive won't mount reformatted volume

Bug #1825596 reported by Jason Zions on 2019年04月20日

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
cloud-init	Fix Released	High	Unassigned

Bug Description

If an Azure VM is rebooted after being moved to a different host (e.g. after a deallocate operation or after a service-heal to remove a bad host from service), the ephemeral drive exposed to the VM is reset to the default state (NTFS format). The Azure data source detects this and marks cc_disk_setup and cc_mounts to be run. While cc_disk_setup reformats the volume as desired, cc_mounts determines that the appropriate mount request was already in /etc/fstab (as setup during initial provisioning). Since the normal boot process would already have mounted everything according to fstab, the cc_mounts logic is "no mount -a is required". This is not true in this scenario.

Related branches

~chad.smith/cloud-init:ubuntu/bionic

Merged into cloud-init:ubuntu/bionic at revision 01cf9304e4d697cffff7db9db48e374b31cb50bd

Ryan Harper: Approve on 2019年05月14日

Server Team CI bot: Approve (continuous-integration) on 2019年05月11日

Diff: 4639 lines (+2418/-613)

49 files modified

ChangeLog (+117/-0)
cloudinit/cmd/main.py (+5/-4)
cloudinit/config/cc_apt_configure.py (+1/-1)
cloudinit/config/cc_mounts.py (+11/-0)
cloudinit/config/cc_ubuntu_advantage.py (+116/-109)
cloudinit/config/cc_ubuntu_drivers.py (+112/-0)
cloudinit/config/tests/test_ubuntu_advantage.py (+191/-156)
cloudinit/config/tests/test_ubuntu_drivers.py (+174/-0)
cloudinit/net/eni.py (+11/-5)
cloudinit/net/network_state.py (+33/-8)
cloudinit/net/sysconfig.py (+29/-11)
cloudinit/net/tests/test_init.py (+1/-1)
cloudinit/reporting/handlers.py (+57/-60)
cloudinit/sources/DataSourceAzure.py (+179/-95)
cloudinit/sources/DataSourceCloudStack.py (+1/-1)
cloudinit/sources/DataSourceConfigDrive.py (+2/-5)
cloudinit/sources/DataSourceEc2.py (+7/-3)
cloudinit/sources/DataSourceNoCloud.py (+3/-1)
cloudinit/sources/helpers/azure.py (+42/-3)
cloudinit/util.py (+17/-13)
cloudinit/version.py (+1/-1)
config/cloud.cfg.tmpl (+3/-0)
debian/changelog (+48/-0)
debian/patches/series (+1/-0)
debian/patches/ubuntu-advantage-revert-tip.patch (+735/-0)
doc/rtd/topics/datasources/azure.rst (+35/-22)
doc/rtd/topics/datasources/nocloud.rst (+1/-1)
doc/rtd/topics/modules.rst (+1/-0)
packages/redhat/cloud-init.spec.in (+3/-1)
packages/suse/cloud-init.spec.in (+3/-1)
setup.py (+2/-1)
tests/cloud_tests/releases.yaml (+16/-0)
tests/cloud_tests/testcases/modules/apt_pipelining_disable.yaml (+1/-2)
tests/cloud_tests/testcases/modules/apt_pipelining_os.py (+3/-3)
tests/cloud_tests/testcases/modules/apt_pipelining_os.yaml (+4/-5)
tests/data/azure/non_unicode_random_string (+1/-0)
tests/unittests/test_datasource/test_azure.py (+32/-5)
tests/unittests/test_datasource/test_azure_helper.py (+7/-2)
tests/unittests/test_datasource/test_nocloud.py (+42/-0)
tests/unittests/test_distros/test_netconfig.py (+2/-0)
tests/unittests/test_ds_identify.py (+17/-0)
tests/unittests/test_handler/test_handler_mounts.py (+29/-1)
tests/unittests/test_handler/test_schema.py (+1/-0)
tests/unittests/test_net.py (+251/-18)
tests/unittests/test_reporting_hyperv.py (+49/-55)
tools/build-on-freebsd (+4/-5)
tools/ds-identify (+4/-3)
tools/read-version (+5/-2)
tox.ini (+8/-9)

api

~chad.smith/cloud-init:ubuntu/cosmic

Merged into cloud-init:ubuntu/cosmic at revision a452dbb7ee4521f3fd730a82fa44b080f60c7839

Ryan Harper: Approve on 2019年05月14日

Server Team CI bot: Approve (continuous-integration) on 2019年05月13日

Diff: 4637 lines (+2416/-613)

49 files modified

ChangeLog (+117/-0)
cloudinit/cmd/main.py (+5/-4)
cloudinit/config/cc_apt_configure.py (+1/-1)
cloudinit/config/cc_mounts.py (+11/-0)
cloudinit/config/cc_ubuntu_advantage.py (+116/-109)
cloudinit/config/cc_ubuntu_drivers.py (+112/-0)
cloudinit/config/tests/test_ubuntu_advantage.py (+191/-156)
cloudinit/config/tests/test_ubuntu_drivers.py (+174/-0)
cloudinit/net/eni.py (+11/-5)
cloudinit/net/network_state.py (+33/-8)
cloudinit/net/sysconfig.py (+29/-11)
cloudinit/net/tests/test_init.py (+1/-1)
cloudinit/reporting/handlers.py (+57/-60)
cloudinit/sources/DataSourceAzure.py (+179/-95)
cloudinit/sources/DataSourceCloudStack.py (+1/-1)
cloudinit/sources/DataSourceConfigDrive.py (+2/-5)
cloudinit/sources/DataSourceEc2.py (+7/-3)
cloudinit/sources/DataSourceNoCloud.py (+3/-1)
cloudinit/sources/helpers/azure.py (+42/-3)
cloudinit/util.py (+17/-13)
cloudinit/version.py (+1/-1)
config/cloud.cfg.tmpl (+3/-0)
debian/changelog (+48/-0)
debian/patches/series (+1/-0)
debian/patches/ubuntu-advantage-revert-tip.patch (+733/-0)
doc/rtd/topics/datasources/azure.rst (+35/-22)
doc/rtd/topics/datasources/nocloud.rst (+1/-1)
doc/rtd/topics/modules.rst (+1/-0)
packages/redhat/cloud-init.spec.in (+3/-1)
packages/suse/cloud-init.spec.in (+3/-1)
setup.py (+2/-1)
tests/cloud_tests/releases.yaml (+16/-0)
tests/cloud_tests/testcases/modules/apt_pipelining_disable.yaml (+1/-2)
tests/cloud_tests/testcases/modules/apt_pipelining_os.py (+3/-3)
tests/cloud_tests/testcases/modules/apt_pipelining_os.yaml (+4/-5)
tests/data/azure/non_unicode_random_string (+1/-0)
tests/unittests/test_datasource/test_azure.py (+32/-5)
tests/unittests/test_datasource/test_azure_helper.py (+7/-2)
tests/unittests/test_datasource/test_nocloud.py (+42/-0)
tests/unittests/test_distros/test_netconfig.py (+2/-0)
tests/unittests/test_ds_identify.py (+17/-0)
tests/unittests/test_handler/test_handler_mounts.py (+29/-1)
tests/unittests/test_handler/test_schema.py (+1/-0)
tests/unittests/test_net.py (+251/-18)
tests/unittests/test_reporting_hyperv.py (+49/-55)
tools/build-on-freebsd (+4/-5)
tools/ds-identify (+4/-3)
tools/read-version (+5/-2)
tox.ini (+8/-9)

api

~chad.smith/cloud-init:ubuntu/disco

Merged into cloud-init:ubuntu/disco at revision 59835f9211e70101bde10c63c7e84cc6a74c930c

Server Team CI bot: Approve (continuous-integration) on 2019年05月11日

cloud-init Commiters: Pending requested 2019年05月11日

Diff: 1142 lines (+417/-168)

25 files modified

ChangeLog (+117/-0)
cloudinit/config/cc_apt_configure.py (+1/-1)
cloudinit/config/cc_mounts.py (+11/-0)
cloudinit/net/sysconfig.py (+4/-2)
cloudinit/net/tests/test_init.py (+1/-1)
cloudinit/reporting/handlers.py (+57/-60)
cloudinit/sources/DataSourceAzure.py (+11/-6)
cloudinit/sources/DataSourceCloudStack.py (+1/-1)
cloudinit/sources/DataSourceConfigDrive.py (+2/-5)
cloudinit/sources/DataSourceEc2.py (+1/-1)
cloudinit/sources/helpers/azure.py (+11/-3)
cloudinit/util.py (+2/-13)
cloudinit/version.py (+1/-1)
debian/changelog (+27/-0)
packages/redhat/cloud-init.spec.in (+3/-1)
packages/suse/cloud-init.spec.in (+3/-1)
setup.py (+2/-1)
tests/cloud_tests/releases.yaml (+16/-0)
tests/unittests/test_datasource/test_azure.py (+10/-3)
tests/unittests/test_datasource/test_azure_helper.py (+7/-2)
tests/unittests/test_handler/test_handler_mounts.py (+29/-1)
tests/unittests/test_net.py (+42/-3)
tests/unittests/test_reporting_hyperv.py (+49/-55)
tools/build-on-freebsd (+4/-5)
tools/read-version (+5/-2)

api

~jasonzio/cloud-init:requestMountAll

Merged into cloud-init:master

Server Team CI bot: Approve (continuous-integration) on 2019年05月08日

Ryan Harper: Approve on 2019年05月08日

Diff: 87 lines (+40/-1)

2 files modified

cloudinit/config/cc_mounts.py (+11/-0)
tests/unittests/test_handler/test_handler_mounts.py (+29/-1)

api

Revision history for this message

Ryan Harper (raharper) wrote on 2019年04月22日: #1

Could you supply the systemd journal from the system and a cloud-init collect-logs?

I'd like to confirm that the .mount units for the ephemeral drive were attempted, but failed due to the disk not being ext4 (yet).

Changed in cloud-init:

importance: Undecided → High

status: New → Incomplete

Revision history for this message

Jason Zions (jasonzio) wrote on 2019年04月22日: #2

I confirmed that myself in the logs; the timestamp on the failed mnt.mount is about three seconds earlier than the timestamp on the cloud-init startup message. The error message is indeed a complaint about not being able to mount an "ntfs" formatted volume. I'll try to find a relevant repro and get you the logs you seek.

Revision history for this message

Jason Zions (jasonzio) wrote on 2019年04月23日: #3

cloud-init.tar.gz Edit (48.6 KiB, application/x-tar)

Per your request, the results of collect-logs

Revision history for this message

Jason Zions (jasonzio) wrote on 2019年04月23日: #4

messages.gz Edit (16.4 KiB, application/octet-stream)

The contents of /var/log/messages on this CentOS system, corresponding to the VM start-after-deallocate operation covered by the previously-uploaded collect-logs bundle. In particular, see Apr 23 00:06:05 (date/times in UTC) for the attempt to mount the NTFS ephemeral volume. Meanwhile, in cloud-init.log, you'll see that cc_disk_setup doesn't run until 00:06:27.

When this VM was first created, /var/log/messages said nothing at all about /dev/disk/cloud/azure_resource-part1 or about NTFS, which is exactly as one would expect for the first-time-ever boot of the VM.

Jason Zions (jasonzio) on 2019年04月23日

Changed in cloud-init:

status: Incomplete → New

Ryan Harper (raharper) on 2019年04月26日

Changed in cloud-init:

status: New → In Progress

Revision history for this message

Server Team CI bot (server-team-bot) wrote on 2019年05月08日: #5

This bug is fixed with commit acc25d8d to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=acc25d8d

Changed in cloud-init:

status: In Progress → Fix Committed

Revision history for this message

Chad Smith (chad.smith) wrote on 2019年05月10日: Fixed in cloud-init version 19.1. #6

This bug is believed to be fixed in cloud-init in version 19.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:

status: Fix Committed → Fix Released

Roufique Hossain (roufique) on 2020年04月21日

Changed in cloud-init:

assignee: nobody → Roufique Hossain (roufique)

Steve Langasek (vorlon) on 2020年04月21日

Changed in cloud-init:

assignee: Roufique Hossain (roufique) → nobody

Revision history for this message

James Falcon (falcojr) wrote on 2023年05月11日: #7

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3370