cloud-init fails to fetch metadata with OpenStack

Bug #894279 reported by Juerg Haefliger
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
High
Scott Moser
cloud-init (Ubuntu)
Fix Released
Low
Unassigned

Bug Description

Tested the UEC Lucid, Maverick, Natty and Oneiric images with OpenStack Diablo. When launching multiple (20+) instances at the same time the OpenStack metadata server becomes quite slow and cloud-init fails to fetch the metadata due to its short timeout of 2 (3 for Oneiric) seconds in DataSourceEc2.py. This is an unreasonably small timeout value. I'd suggest a timeout of 60 secs and 5 retries as the default.

init: cloud-init-local main process (217) terminated with status 1
init: cloud-init-nonet main process (290) killed by TERM signal
init: udev-fallback-graphics main process (317) terminated with status 1
init: plymouth main process (59) killed by SEGV signal
init: plymouth-splash main process (345) terminated with status 2
cloud-init start running: 2011年9月30日 12:01:48 +0000. up 2.17 seconds
2011年09月30日 12:01:50,874 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id
2011年09月30日 12:01:50,875 - DataSourceEc2.py[WARNING]: 12:01:50 [ 1/30]: url error [timed out]
2011年09月30日 12:01:53,879 - DataSourceEc2.py[WARNING]: 12:01:53 [ 2/30]: url error [timed out]
2011年09月30日 12:01:54,883 - DataSourceEc2.py[WARNING]: 12:01:54 [ 3/30]: url error [[Errno 113] No route to host]
2011年09月30日 12:01:57,887 - DataSourceEc2.py[WARNING]: 12:01:57 [ 4/30]: url error [timed out]
2011年09月30日 12:01:58,893 - DataSourceEc2.py[WARNING]: 12:01:58 [ 5/30]: url error [[Errno 113] No route to host]
2011年09月30日 12:02:01,897 - DataSourceEc2.py[WARNING]: 12:02:01 [ 6/30]: url error [timed out]
2011年09月30日 12:02:05,902 - DataSourceEc2.py[WARNING]: 12:02:05 [ 7/30]: url error [timed out]
2011年09月30日 12:02:09,907 - DataSourceEc2.py[WARNING]: 12:02:09 [ 8/30]: url error [timed out]
2011年09月30日 12:02:13,913 - DataSourceEc2.py[WARNING]: 12:02:13 [ 9/30]: url error [timed out]
2011年09月30日 12:02:17,918 - DataSourceEc2.py[WARNING]: 12:02:17 [10/30]: url error [timed out]
2011年09月30日 12:02:21,923 - DataSourceEc2.py[WARNING]: 12:02:21 [11/30]: url error [timed out]
2011年09月30日 12:02:26,773 - DataSourceEc2.py[WARNING]: 12:02:26 [12/30]: url error [[Errno 113] No route to host]
2011年09月30日 12:02:31,778 - DataSourceEc2.py[WARNING]: 12:02:31 [13/30]: url error [timed out]
2011年09月30日 12:02:36,785 - DataSourceEc2.py[WARNING]: 12:02:36 [14/30]: url error [timed out]
2011年09月30日 12:02:41,791 - DataSourceEc2.py[WARNING]: 12:02:41 [15/30]: url error [timed out]
2011年09月30日 12:02:46,797 - DataSourceEc2.py[WARNING]: 12:02:46 [16/30]: url error [timed out]
2011年09月30日 12:02:52,013 - DataSourceEc2.py[WARNING]: 12:02:52 [17/30]: url error [[Errno 113] No route to host]
2011年09月30日 12:02:58,019 - DataSourceEc2.py[WARNING]: 12:02:58 [18/30]: url error [timed out]
2011年09月30日 12:03:04,026 - DataSourceEc2.py[WARNING]: 12:03:04 [19/30]: url error [timed out]
2011年09月30日 12:03:10,033 - DataSourceEc2.py[WARNING]: 12:03:10 [20/30]: url error [timed out]
2011年09月30日 12:03:16,040 - DataSourceEc2.py[WARNING]: 12:03:16 [21/30]: url error [timed out]
2011年09月30日 12:03:23,023 - DataSourceEc2.py[WARNING]: 12:03:23 [22/30]: url error [[Errno 113] No route to host]
2011年09月30日 12:03:30,030 - DataSourceEc2.py[WARNING]: 12:03:30 [23/30]: url error [timed out]
2011年09月30日 12:03:37,038 - DataSourceEc2.py[WARNING]: 12:03:37 [24/30]: url error [timed out]
2011年09月30日 12:03:44,046 - DataSourceEc2.py[WARNING]: 12:03:44 [25/30]: url error [timed out]
2011年09月30日 12:03:51,054 - DataSourceEc2.py[WARNING]: 12:03:51 [26/30]: url error [timed out]
2011年09月30日 12:03:59,063 - DataSourceEc2.py[WARNING]: 12:03:59 [27/30]: url error [timed out]
2011年09月30日 12:04:07,071 - DataSourceEc2.py[WARNING]: 12:04:07 [28/30]: url error [timed out]
2011年09月30日 12:04:15,082 - DataSourceEc2.py[WARNING]: 12:04:15 [29/30]: url error [timed out]
2011年09月30日 12:04:23,091 - DataSourceEc2.py[WARNING]: 12:04:23 [30/30]: url error [timed out]
2011年09月30日 12:04:29,097 - DataSourceEc2.py[CRITICAL]: giving up on md after 160 seconds

no instance data found in start

Related branches

lp:~cloud-init-dev/cloud-init/trunk
Chad Smith (community): Needs Fixing
Hannes: Pending requested
Diff: 51583 lines (+11/-49380) (has conflicts)
424 files modified
.bzrignore (+0/-4)
ChangeLog (+0/-761)
HACKING.rst (+0/-48)
LICENSE (+0/-674)
MANIFEST.in (+0/-8)
Makefile (+0/-82)
README (+11/-0)
TODO.rst (+0/-43)
bin/cloud-init (+0/-673)
cloudinit/__init__.py (+0/-21)
cloudinit/cloud.py (+0/-109)
cloudinit/config/__init__.py (+0/-58)
cloudinit/config/cc_apt_configure.py (+0/-292)
cloudinit/config/cc_apt_pipelining.py (+0/-57)
cloudinit/config/cc_bootcmd.py (+0/-54)
cloudinit/config/cc_byobu.py (+0/-80)
cloudinit/config/cc_ca_certs.py (+0/-104)
cloudinit/config/cc_chef.py (+0/-342)
cloudinit/config/cc_debug.py (+0/-109)
cloudinit/config/cc_disable_ec2_metadata.py (+0/-36)
cloudinit/config/cc_emit_upstart.py (+0/-69)
cloudinit/config/cc_fan.py (+0/-101)
cloudinit/config/cc_final_message.py (+0/-73)
cloudinit/config/cc_foo.py (+0/-52)
cloudinit/config/cc_growpart.py (+0/-300)
cloudinit/config/cc_grub_dpkg.py (+0/-73)
cloudinit/config/cc_keys_to_console.py (+0/-62)
cloudinit/config/cc_landscape.py (+0/-99)
cloudinit/config/cc_locale.py (+0/-37)
cloudinit/config/cc_lxd.py (+0/-176)
cloudinit/config/cc_mcollective.py (+0/-88)
cloudinit/config/cc_migrator.py (+0/-85)
cloudinit/config/cc_mounts.py (+0/-405)
cloudinit/config/cc_package_update_upgrade_install.py (+0/-99)
cloudinit/config/cc_phone_home.py (+0/-122)
cloudinit/config/cc_power_state_change.py (+0/-223)
cloudinit/config/cc_puppet.py (+0/-118)
cloudinit/config/cc_resizefs.py (+0/-185)
cloudinit/config/cc_resolv_conf.py (+0/-116)
cloudinit/config/cc_rh_subscription.py (+0/-408)
cloudinit/config/cc_rightscale_userdata.py (+0/-102)
cloudinit/config/cc_rsyslog.py (+0/-366)
cloudinit/config/cc_runcmd.py (+0/-38)
cloudinit/config/cc_salt_minion.py (+0/-59)
cloudinit/config/cc_scripts_per_boot.py (+0/-41)
cloudinit/config/cc_scripts_per_instance.py (+0/-41)
cloudinit/config/cc_scripts_per_once.py (+0/-41)
cloudinit/config/cc_scripts_user.py (+0/-42)
cloudinit/config/cc_scripts_vendor.py (+0/-43)
cloudinit/config/cc_seed_random.py (+0/-94)
cloudinit/config/cc_set_hostname.py (+0/-37)
cloudinit/config/cc_set_passwords.py (+0/-167)
cloudinit/config/cc_snappy.py (+0/-304)
cloudinit/config/cc_ssh.py (+0/-142)
cloudinit/config/cc_ssh_authkey_fingerprints.py (+0/-105)
cloudinit/config/cc_ssh_import_id.py (+0/-99)
cloudinit/config/cc_timezone.py (+0/-39)
cloudinit/config/cc_ubuntu_init_switch.py (+0/-162)
cloudinit/config/cc_update_etc_hosts.py (+0/-60)
cloudinit/config/cc_update_hostname.py (+0/-43)
cloudinit/config/cc_users_groups.py (+0/-34)
cloudinit/config/cc_write_files.py (+0/-103)
cloudinit/config/cc_yum_add_repo.py (+0/-107)
cloudinit/cs_utils.py (+0/-105)
cloudinit/distros/__init__.py (+0/-950)
cloudinit/distros/arch.py (+0/-201)
cloudinit/distros/debian.py (+0/-227)
cloudinit/distros/fedora.py (+0/-31)
cloudinit/distros/freebsd.py (+0/-417)
cloudinit/distros/gentoo.py (+0/-160)
cloudinit/distros/net_util.py (+0/-182)
cloudinit/distros/parsers/__init__.py (+0/-28)
cloudinit/distros/parsers/hostname.py (+0/-88)
cloudinit/distros/parsers/hosts.py (+0/-92)
cloudinit/distros/parsers/resolv_conf.py (+0/-169)
cloudinit/distros/parsers/sys_conf.py (+0/-113)
cloudinit/distros/rhel.py (+0/-222)
cloudinit/distros/rhel_util.py (+0/-89)
cloudinit/distros/sles.py (+0/-179)
cloudinit/distros/ubuntu.py (+0/-31)
cloudinit/ec2_utils.py (+0/-201)
cloudinit/filters/__init__.py (+0/-21)
cloudinit/filters/launch_index.py (+0/-75)
cloudinit/handlers/__init__.py (+0/-274)
cloudinit/handlers/boot_hook.py (+0/-70)
cloudinit/handlers/cloud_config.py (+0/-163)
cloudinit/handlers/shell_script.py (+0/-55)
cloudinit/handlers/upstart_job.py (+0/-119)
cloudinit/helpers.py (+0/-458)
cloudinit/importer.py (+0/-58)
cloudinit/log.py (+0/-155)
cloudinit/mergers/__init__.py (+0/-166)
cloudinit/mergers/m_dict.py (+0/-88)
cloudinit/mergers/m_list.py (+0/-89)
cloudinit/mergers/m_str.py (+0/-46)
cloudinit/net/__init__.py (+0/-771)
cloudinit/net/network_state.py (+0/-447)
cloudinit/net/udev.py (+0/-54)
cloudinit/netinfo.py (+0/-249)
cloudinit/patcher.py (+0/-58)
cloudinit/registry.py (+0/-37)
cloudinit/reporting/__init__.py (+0/-42)
cloudinit/reporting/events.py (+0/-248)
cloudinit/reporting/handlers.py (+0/-91)
cloudinit/safeyaml.py (+0/-32)
cloudinit/settings.py (+0/-68)
cloudinit/signal_handler.py (+0/-71)
cloudinit/sources/DataSourceAltCloud.py (+0/-293)
cloudinit/sources/DataSourceAzure.py (+0/-651)
cloudinit/sources/DataSourceBigstep.py (+0/-57)
cloudinit/sources/DataSourceCloudSigma.py (+0/-141)
cloudinit/sources/DataSourceCloudStack.py (+0/-253)
cloudinit/sources/DataSourceConfigDrive.py (+0/-424)
cloudinit/sources/DataSourceDigitalOcean.py (+0/-110)
cloudinit/sources/DataSourceEc2.py (+0/-211)
cloudinit/sources/DataSourceGCE.py (+0/-167)
cloudinit/sources/DataSourceMAAS.py (+0/-353)
cloudinit/sources/DataSourceNoCloud.py (+0/-336)
cloudinit/sources/DataSourceNone.py (+0/-57)
cloudinit/sources/DataSourceOVF.py (+0/-429)
cloudinit/sources/DataSourceOpenNebula.py (+0/-455)
cloudinit/sources/DataSourceOpenStack.py (+0/-173)
cloudinit/sources/DataSourceSmartOS.py (+0/-536)
cloudinit/sources/__init__.py (+0/-337)
cloudinit/sources/helpers/__init__.py (+0/-13)
cloudinit/sources/helpers/azure.py (+0/-279)
cloudinit/sources/helpers/openstack.py (+0/-499)
cloudinit/sources/helpers/vmware/__init__.py (+0/-13)
cloudinit/sources/helpers/vmware/imc/__init__.py (+0/-13)
cloudinit/sources/helpers/vmware/imc/boot_proto.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/config.py (+0/-95)
cloudinit/sources/helpers/vmware/imc/config_file.py (+0/-129)
cloudinit/sources/helpers/vmware/imc/config_namespace.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/config_nic.py (+0/-247)
cloudinit/sources/helpers/vmware/imc/config_source.py (+0/-23)
cloudinit/sources/helpers/vmware/imc/guestcust_error.py (+0/-24)
cloudinit/sources/helpers/vmware/imc/guestcust_event.py (+0/-27)
cloudinit/sources/helpers/vmware/imc/guestcust_state.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/guestcust_util.py (+0/-128)
cloudinit/sources/helpers/vmware/imc/ipv4_mode.py (+0/-45)
cloudinit/sources/helpers/vmware/imc/nic.py (+0/-147)
cloudinit/sources/helpers/vmware/imc/nic_base.py (+0/-154)
cloudinit/ssh_util.py (+0/-314)
cloudinit/stages.py (+0/-844)
cloudinit/templater.py (+0/-149)
cloudinit/type_utils.py (+0/-52)
cloudinit/url_helper.py (+0/-509)
cloudinit/user_data.py (+0/-352)
cloudinit/util.py (+0/-2226)
cloudinit/version.py (+0/-27)
config/cloud.cfg (+0/-115)
config/cloud.cfg-freebsd (+0/-88)
config/cloud.cfg.d/05_logging.cfg (+0/-66)
config/cloud.cfg.d/README (+0/-3)
doc/README (+0/-4)
doc/examples/cloud-config-add-apt-repos.txt (+0/-34)
doc/examples/cloud-config-archive-launch-index.txt (+0/-30)
doc/examples/cloud-config-archive.txt (+0/-16)
doc/examples/cloud-config-boot-cmds.txt (+0/-15)
doc/examples/cloud-config-ca-certs.txt (+0/-31)
doc/examples/cloud-config-chef-oneiric.txt (+0/-90)
doc/examples/cloud-config-chef.txt (+0/-95)
doc/examples/cloud-config-datasources.txt (+0/-73)
doc/examples/cloud-config-disk-setup.txt (+0/-251)
doc/examples/cloud-config-final-message.txt (+0/-7)
doc/examples/cloud-config-gluster.txt (+0/-18)
doc/examples/cloud-config-growpart.txt (+0/-31)
doc/examples/cloud-config-install-packages.txt (+0/-15)
doc/examples/cloud-config-landscape.txt (+0/-22)
doc/examples/cloud-config-launch-index.txt (+0/-23)
doc/examples/cloud-config-lxd.txt (+0/-55)
doc/examples/cloud-config-mcollective.txt (+0/-49)
doc/examples/cloud-config-mount-points.txt (+0/-46)
doc/examples/cloud-config-phone-home.txt (+0/-14)
doc/examples/cloud-config-power-state.txt (+0/-40)
doc/examples/cloud-config-puppet.txt (+0/-51)
doc/examples/cloud-config-reporting.txt (+0/-17)
doc/examples/cloud-config-resolv-conf.txt (+0/-20)
doc/examples/cloud-config-rh_subscription.txt (+0/-49)
doc/examples/cloud-config-rsyslog.txt (+0/-46)
doc/examples/cloud-config-run-cmds.txt (+0/-22)
doc/examples/cloud-config-salt-minion.txt (+0/-53)
doc/examples/cloud-config-seed-random.txt (+0/-32)
doc/examples/cloud-config-ssh-keys.txt (+0/-46)
doc/examples/cloud-config-update-apt.txt (+0/-7)
doc/examples/cloud-config-update-packages.txt (+0/-8)
doc/examples/cloud-config-user-groups.txt (+0/-109)
doc/examples/cloud-config-vendor-data.txt (+0/-16)
doc/examples/cloud-config-write-files.txt (+0/-33)
doc/examples/cloud-config-yum-repo.txt (+0/-20)
doc/examples/cloud-config.txt (+0/-645)
doc/examples/include-once.txt (+0/-7)
doc/examples/include.txt (+0/-5)
doc/examples/kernel-cmdline.txt (+0/-18)
doc/examples/part-handler-v2.txt (+0/-38)
doc/examples/part-handler.txt (+0/-23)
doc/examples/plain-ignored.txt (+0/-2)
doc/examples/seed/README (+0/-22)
doc/examples/seed/meta-data (+0/-30)
doc/examples/seed/user-data (+0/-3)
doc/examples/upstart-cloud-config.txt (+0/-12)
doc/examples/upstart-rclocal.txt (+0/-12)
doc/examples/user-script.txt (+0/-8)
doc/merging.rst (+0/-188)
doc/rtd/conf.py (+0/-77)
doc/rtd/index.rst (+0/-31)
doc/rtd/static/logo.svg (+0/-89)
doc/rtd/topics/availability.rst (+0/-20)
doc/rtd/topics/capabilities.rst (+0/-24)
doc/rtd/topics/datasources.rst (+0/-200)
doc/rtd/topics/dir_layout.rst (+0/-81)
doc/rtd/topics/examples.rst (+0/-133)
doc/rtd/topics/format.rst (+0/-159)
doc/rtd/topics/hacking.rst (+0/-1)
doc/rtd/topics/merging.rst (+0/-5)
doc/rtd/topics/modules.rst (+0/-342)
doc/rtd/topics/moreinfo.rst (+0/-12)
doc/sources/altcloud/README.rst (+0/-87)
doc/sources/azure/README.rst (+0/-134)
doc/sources/cloudsigma/README.rst (+0/-38)
doc/sources/cloudstack/README.rst (+0/-29)
doc/sources/configdrive/README.rst (+0/-123)
doc/sources/digitalocean/README.rst (+0/-21)
doc/sources/kernel-cmdline.txt (+0/-48)
doc/sources/nocloud/README.rst (+0/-71)
doc/sources/opennebula/README.rst (+0/-142)
doc/sources/openstack/README.rst (+0/-24)
doc/sources/ovf/README (+0/-83)
doc/sources/ovf/example/ovf-env.xml (+0/-46)
doc/sources/ovf/example/ubuntu-server.ovf (+0/-130)
doc/sources/ovf/make-iso (+0/-156)
doc/sources/ovf/ovf-env.xml.tmpl (+0/-28)
doc/sources/ovf/ovfdemo.pem (+0/-27)
doc/sources/ovf/user-data (+0/-7)
doc/sources/smartos/README.rst (+0/-149)
doc/status.txt (+0/-53)
doc/userdata.txt (+0/-79)
doc/var-lib-cloud.txt (+0/-63)
doc/vendordata.txt (+0/-53)
packages/bddeb (+0/-260)
packages/brpm (+0/-277)
packages/debian/changelog.in (+0/-6)
packages/debian/cloud-init.postinst (+0/-16)
packages/debian/cloud-init.preinst (+0/-20)
packages/debian/compat (+0/-1)
packages/debian/control.in (+0/-29)
packages/debian/copyright (+0/-29)
packages/debian/dirs (+0/-6)
packages/debian/rules.in (+0/-23)
packages/debian/watch (+0/-2)
packages/redhat/cloud-init.spec.in (+0/-200)
packages/suse/cloud-init.spec.in (+0/-163)
requirements.txt (+0/-36)
setup.py (+0/-215)
systemd/cloud-config.service (+0/-16)
systemd/cloud-config.target (+0/-11)
systemd/cloud-final.service (+0/-17)
systemd/cloud-init-generator (+0/-133)
systemd/cloud-init-local.service (+0/-22)
systemd/cloud-init.service (+0/-18)
systemd/cloud-init.target (+0/-6)
sysvinit/debian/cloud-config (+0/-64)
sysvinit/debian/cloud-final (+0/-66)
sysvinit/debian/cloud-init (+0/-64)
sysvinit/debian/cloud-init-local (+0/-63)
sysvinit/freebsd/cloudconfig (+0/-35)
sysvinit/freebsd/cloudfinal (+0/-35)
sysvinit/freebsd/cloudinit (+0/-35)
sysvinit/freebsd/cloudinitlocal (+0/-35)
sysvinit/gentoo/cloud-config (+0/-13)
sysvinit/gentoo/cloud-final (+0/-11)
sysvinit/gentoo/cloud-init (+0/-12)
sysvinit/gentoo/cloud-init-local (+0/-13)
sysvinit/redhat/cloud-config (+0/-121)
sysvinit/redhat/cloud-final (+0/-121)
sysvinit/redhat/cloud-init (+0/-121)
sysvinit/redhat/cloud-init-local (+0/-124)
templates/chef_client.rb.tmpl (+0/-58)
templates/hosts.debian.tmpl (+0/-26)
templates/hosts.freebsd.tmpl (+0/-24)
templates/hosts.redhat.tmpl (+0/-24)
templates/hosts.suse.tmpl (+0/-26)
templates/resolv.conf.tmpl (+0/-30)
templates/sources.list.debian.tmpl (+0/-32)
templates/sources.list.ubuntu.tmpl (+0/-58)
test-requirements.txt (+0/-17)
tests/configs/sample1.yaml (+0/-52)
tests/data/filter_cloud_multipart.yaml (+0/-30)
tests/data/filter_cloud_multipart_1.email (+0/-11)
tests/data/filter_cloud_multipart_2.email (+0/-39)
tests/data/filter_cloud_multipart_header.email (+0/-11)
tests/data/merge_sources/expected1.yaml (+0/-1)
tests/data/merge_sources/expected10.yaml (+0/-7)
tests/data/merge_sources/expected11.yaml (+0/-5)
tests/data/merge_sources/expected12.yaml (+0/-5)
tests/data/merge_sources/expected2.yaml (+0/-3)
tests/data/merge_sources/expected3.yaml (+0/-1)
tests/data/merge_sources/expected4.yaml (+0/-2)
tests/data/merge_sources/expected5.yaml (+0/-7)
tests/data/merge_sources/expected6.yaml (+0/-9)
tests/data/merge_sources/expected7.yaml (+0/-38)
tests/data/merge_sources/expected8.yaml (+0/-7)
tests/data/merge_sources/expected9.yaml (+0/-5)
tests/data/merge_sources/source1-1.yaml (+0/-3)
tests/data/merge_sources/source1-2.yaml (+0/-5)
tests/data/merge_sources/source10-1.yaml (+0/-6)
tests/data/merge_sources/source10-2.yaml (+0/-6)
tests/data/merge_sources/source11-1.yaml (+0/-5)
tests/data/merge_sources/source11-2.yaml (+0/-3)
tests/data/merge_sources/source11-3.yaml (+0/-3)
tests/data/merge_sources/source12-1.yaml (+0/-8)
tests/data/merge_sources/source12-2.yaml (+0/-5)
tests/data/merge_sources/source2-1.yaml (+0/-6)
tests/data/merge_sources/source2-2.yaml (+0/-5)
tests/data/merge_sources/source3-1.yaml (+0/-4)
tests/data/merge_sources/source3-2.yaml (+0/-4)
tests/data/merge_sources/source4-1.yaml (+0/-3)
tests/data/merge_sources/source4-2.yaml (+0/-6)
tests/data/merge_sources/source5-1.yaml (+0/-6)
tests/data/merge_sources/source5-2.yaml (+0/-8)
tests/data/merge_sources/source6-1.yaml (+0/-5)
tests/data/merge_sources/source6-2.yaml (+0/-8)
tests/data/merge_sources/source7-1.yaml (+0/-27)
tests/data/merge_sources/source7-2.yaml (+0/-17)
tests/data/merge_sources/source8-1.yaml (+0/-7)
tests/data/merge_sources/source8-2.yaml (+0/-6)
tests/data/merge_sources/source9-1.yaml (+0/-5)
tests/data/merge_sources/source9-2.yaml (+0/-6)
tests/data/mountinfo_precise_ext4.txt (+0/-24)
tests/data/mountinfo_raring_btrfs.txt (+0/-13)
tests/data/roots/simple_ubuntu/etc/networks/interfaces (+0/-3)
tests/data/user_data.1.txt (+0/-15)
tests/data/vmware/cust-dhcp-2nic.cfg (+0/-34)
tests/data/vmware/cust-static-2nic.cfg (+0/-39)
tests/unittests/helpers.py (+0/-346)
tests/unittests/test__init__.py (+0/-220)
tests/unittests/test_builtin_handlers.py (+0/-73)
tests/unittests/test_cli.py (+0/-54)
tests/unittests/test_cs_util.py (+0/-84)
tests/unittests/test_data.py (+0/-559)
tests/unittests/test_datasource/test_altcloud.py (+0/-452)
tests/unittests/test_datasource/test_azure.py (+0/-646)
tests/unittests/test_datasource/test_azure_helper.py (+0/-421)
tests/unittests/test_datasource/test_cloudsigma.py (+0/-99)
tests/unittests/test_datasource/test_cloudstack.py (+0/-87)
tests/unittests/test_datasource/test_configdrive.py (+0/-407)
tests/unittests/test_datasource/test_digitalocean.py (+0/-127)
tests/unittests/test_datasource/test_gce.py (+0/-166)
tests/unittests/test_datasource/test_maas.py (+0/-163)
tests/unittests/test_datasource/test_nocloud.py (+0/-187)
tests/unittests/test_datasource/test_opennebula.py (+0/-300)
tests/unittests/test_datasource/test_openstack.py (+0/-347)
tests/unittests/test_datasource/test_smartos.py (+0/-585)
tests/unittests/test_distros/test_generic.py (+0/-233)
tests/unittests/test_distros/test_hostname.py (+0/-38)
tests/unittests/test_distros/test_hosts.py (+0/-41)
tests/unittests/test_distros/test_netconfig.py (+0/-321)
tests/unittests/test_distros/test_resolv.py (+0/-67)
tests/unittests/test_distros/test_sysconfig.py (+0/-82)
tests/unittests/test_distros/test_user_data_normalize.py (+0/-297)
tests/unittests/test_ec2_util.py (+0/-139)
tests/unittests/test_filters/test_launch_index.py (+0/-132)
tests/unittests/test_handler/test_handler_apt_configure.py (+0/-109)
tests/unittests/test_handler/test_handler_ca_certs.py (+0/-271)
tests/unittests/test_handler/test_handler_chef.py (+0/-192)
tests/unittests/test_handler/test_handler_debug.py (+0/-81)
tests/unittests/test_handler/test_handler_disk_setup.py (+0/-30)
tests/unittests/test_handler/test_handler_growpart.py (+0/-220)
tests/unittests/test_handler/test_handler_locale.py (+0/-67)
tests/unittests/test_handler/test_handler_lxd.py (+0/-134)
tests/unittests/test_handler/test_handler_mounts.py (+0/-133)
tests/unittests/test_handler/test_handler_power_state.py (+0/-127)
tests/unittests/test_handler/test_handler_rsyslog.py (+0/-174)
tests/unittests/test_handler/test_handler_seed_random.py (+0/-227)
tests/unittests/test_handler/test_handler_set_hostname.py (+0/-72)
tests/unittests/test_handler/test_handler_snappy.py (+0/-306)
tests/unittests/test_handler/test_handler_timezone.py (+0/-76)
tests/unittests/test_handler/test_handler_write_files.py (+0/-112)
tests/unittests/test_handler/test_handler_yum_add_repo.py (+0/-68)
tests/unittests/test_helpers.py (+0/-33)
tests/unittests/test_merging.py (+0/-257)
tests/unittests/test_net.py (+0/-130)
tests/unittests/test_pathprefix2dict.py (+0/-44)
tests/unittests/test_registry.py (+0/-28)
tests/unittests/test_reporting.py (+0/-369)
tests/unittests/test_rh_subscription.py (+0/-214)
tests/unittests/test_runs/test_merge_run.py (+0/-54)
tests/unittests/test_runs/test_simple_run.py (+0/-81)
tests/unittests/test_sshutil.py (+0/-171)
tests/unittests/test_templating.py (+0/-119)
tests/unittests/test_util.py (+0/-489)
tests/unittests/test_vmware_config_file.py (+0/-103)
tools/21-cloudinit.conf (+0/-6)
tools/Z99-cloud-locale-test.sh (+0/-98)
tools/build-on-freebsd (+0/-66)
tools/ccfg-merge-debug (+0/-90)
tools/cloud-init-per (+0/-60)
tools/hacking.py (+0/-170)
tools/make-dist-tarball (+0/-21)
tools/make-mime.py (+0/-60)
tools/make-tarball (+0/-39)
tools/mock-meta.py (+0/-454)
tools/motd-hook (+0/-35)
tools/read-dependencies (+0/-29)
tools/read-version (+0/-26)
tools/run-pep8 (+0/-22)
tools/run-pyflakes (+0/-18)
tools/run-pyflakes3 (+0/-2)
tools/tox-venv (+0/-42)
tools/uncloud-init (+0/-141)
tools/validate-yaml.py (+0/-25)
tools/write-ssh-key-fingerprints (+0/-38)
tox.ini (+0/-35)
udev/66-azure-ephemeral.rules (+0/-18)
udev/79-cloud-init-net-wait.rules (+0/-10)
udev/cloud-init-wait (+0/-70)
upstart/cloud-config.conf (+0/-9)
upstart/cloud-final.conf (+0/-10)
upstart/cloud-init-blocknet.conf (+0/-83)
upstart/cloud-init-container.conf (+0/-57)
upstart/cloud-init-local.conf (+0/-16)
upstart/cloud-init-nonet.conf (+0/-66)
upstart/cloud-init.conf (+0/-9)
upstart/cloud-log-shutdown.conf (+0/-19)
api
JJ Asghar: Pending requested
Diff: 0 lines api
Scott Moser: Pending requested
Diff: 3061 lines (+2239/-175)
39 files modified
ChangeLog (+39/-9)
cloud-init.py (+31/-1)
cloudinit/CloudConfig/__init__.py (+1/-1)
cloudinit/CloudConfig/cc_apt_pipelining.py (+53/-0)
cloudinit/CloudConfig/cc_ca_certs.py (+3/-1)
cloudinit/CloudConfig/cc_chef.py (+5/-5)
cloudinit/CloudConfig/cc_landscape.py (+5/-0)
cloudinit/CloudConfig/cc_resizefs.py (+31/-12)
cloudinit/CloudConfig/cc_salt_minion.py (+2/-1)
cloudinit/CloudConfig/cc_update_etc_hosts.py (+1/-1)
cloudinit/DataSource.py (+4/-1)
cloudinit/DataSourceCloudStack.py (+92/-0)
cloudinit/DataSourceConfigDrive.py (+231/-0)
cloudinit/DataSourceEc2.py (+2/-84)
cloudinit/DataSourceMAAS.py (+345/-0)
cloudinit/DataSourceNoCloud.py (+75/-2)
cloudinit/DataSourceOVF.py (+1/-1)
cloudinit/SshUtil.py (+2/-0)
cloudinit/UserDataHandler.py (+2/-2)
cloudinit/__init__.py (+44/-9)
cloudinit/netinfo.py (+5/-5)
cloudinit/util.py (+253/-30)
config/cloud.cfg (+3/-1)
debian.trunk/control (+1/-0)
doc/configdrive/README (+118/-0)
doc/examples/cloud-config-chef-oneiric.txt (+90/-0)
doc/examples/cloud-config-chef.txt (+48/-6)
doc/examples/cloud-config-datasources.txt (+18/-0)
doc/examples/cloud-config.txt (+11/-0)
doc/kernel-cmdline.txt (+48/-0)
doc/nocloud/README (+55/-0)
setup.py (+1/-0)
tests/unittests/test__init__.py (+242/-0)
tests/unittests/test_datasource/test_maas.py (+153/-0)
tests/unittests/test_handler/test_handler_ca_certs.py (+4/-0)
tests/unittests/test_userdata.py (+107/-0)
tests/unittests/test_util.py (+18/-2)
tools/Z99-cloud-locale-test.sh (+92/-0)
tools/run-pylint (+3/-1)
api
Juerg Haefliger (juergh)
affects: cloud-utils (Ubuntu) → cloud-init (Ubuntu)
Revision history for this message
Scott Moser (smoser) wrote :

2 seconds might have been low, I agree, however the following works fine:
$ for s in cnn.com brickies.net slashdot.org espn.com; do
python -c 'import urllib2, sys; req=urllib2.Request(sys.argv[1]); \
resp=urllib2.urlopen(req, timeout=float(sys.argv[2])); resp.read()'
"http://$s" .1
done

That does the same urllib2.urlopen with a timeout that cloud-init does.
Note the '.1' second timeout for all of those sites that are doing loads
more than serving a 10s of bytes of text (ok, brickies.net is not, but it
is also on shared hosting). I had to set that to .08 to get failures on
any of the listed sites. 3 seconds (the default in the code) should be
plenty.

But... we can up it,and it probably was arbitrarily low.

we had to set a timeout otherwise there is none, and due to the fact
that cloud-init retries the MD again and again (giving it time to come
up due to bugs/bad-design in EC2 and Eucalyptus). If the address
169.254.169.254 has been firewalled off, you potentially hang indefinitely
trying to connect (in the case where you're not running in a cloud).

I urge anyone interested in this to help to fix this bug in openstack.
There is a proposed solution at http://pad.lv/851159 although I have not
tested. Additionally, it is suggested (by vish and others) that you can
reduce this slowness by running an API server on each of the nodes. I
don't know how feasible that is. It is simply absurd to not be able to
respond to requests to the metadata service in less than 3 seconds (or 5
or 10).

The issue we were trying to protect against really was "non-cloud"
installation of cloud-init. Ie, in 10.04 or maybe maverick, install of
cloud-init on your laptop, and reboot would be blocked boot by cloud-init
for something like 1500 seconds. No fun.

Regarding your suggestion of 60 second timeout, and 5 retries, that
would also be insufficient. There is a case where the metadata service
isn't ready yet (this occurred on Eucalyptus and we've also seen it on
EC2, although not recently on either). In that case, the connections
would immeditely get "connection refused", so the 5 retries would occur
within 5 seconds, and never find the Metadata service.

Revision history for this message
Juerg Haefliger (juergh) wrote :

I agree that Nova needs to be fixed but we need a cloud-init fix for the current Nova as well. As for running an API server on each compute node, I don't think that's a viable solution for our environment. We've modified memcached to cache the metadata requests and that seems to help make things faster.

I see your point about the metadata service not being ready in time. Do you know how long it can take for the service to become ready? How about a smarter sleeptime that always increases (linearly) from 1 to 20 secs (or some configurable upper limit)?
sleeptime = int(19 * x / (sleeps - 1)) + 1
With that, the 5 retries would happen over a time of 51 secs (1, 5, 10, 15, 20 sec sleeps in between the 5 requests).

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I'll mark this confirmed as I think it's agreed there is aproblem, just not what to do about it.

Changed in cloud-init (Ubuntu):
importance: Undecided → Low
status: New → Confirmed
Revision history for this message
Scott Moser (smoser) wrote :

I'll get to this early next week.
but one more comment. The initial poll that is timing out for openstack is just to check to see if the metadata service is there (ie, "is this EC2?"). Then immediately after that, cloud-init crawls the metadata service. So, we make 1 call until it succedes , and then maybe 18 calls or so. So if its a 10 second wait, that blocks boot by 180 seconds on the crawl :-(

Revision history for this message
Scott Moser (smoser) wrote :

@Juerg,
Could you test out the cloud-init at trunk right now and see if it fixes your problem?
The commit at http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/488 hopefully has a readable description of hte change.

To build a upstream cloud-init, you can just run:
./tools/bddeb

Changed in cloud-init:
assignee: nobody → Scott Moser (smoser)
importance: Undecided → High
status: New → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3~bzr497-0ubuntu1

---------------
cloud-init (0.6.3~bzr497-0ubuntu1) precise; urgency=low

* New upstream snapshot.
- cloud-config support for configuring apt-proxy
- selection of local mirror based on presense of 'ubuntu-mirror' dns
entry in local domain. (LP: #897688)
- DataSourceEc2: more resilliant to slow metadata service (LP: #894279)
- close stdin in all programs launched by cloud-init (LP: #903993)
- revert management of /etc/hosts to 0.6.1 style (LP: #890501, LP: #871966)
- write full ssh keys to console for easy machine consumption (LP: #893400)
- put INSTANCE_ID environment variable in bootcmd scripts
- add 'cloud-init-per' script for easily running things with a given freq
(this replaced cloud-init-run-module)
- support configuration of landscape-client via cloud-config (LP: #857366)
- part-handlers now get base64 decoded content rather than 2xbase64 encoded
in the payload parameter. (LP: #874342)
-- Scott Moser <email address hidden> 2011年12月22日 04:07:38 -0500

Changed in cloud-init (Ubuntu):
status: Confirmed → Fix Released
Scott Moser (smoser)
Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
Craig Sanders (cas) wrote :

Hi, It's great that this is fixed in precise....but has it been back-ported to lucid?

There's going to be a lot of lucid instances running for a long time to come, even after precise is released.

I've built myself a lucid development VM, and tried to build the latest version of cloud-init from precise. it won't build for a number of reasons - starting with lucid's python 2.6 version being too old, and no python-mocker. I've spent several hours on it so far, and wanted to check if i'm just wasting time duplicating work that's already been done before continuing any further.

thanks,

craig

Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

AltStyle によって変換されたページ (->オリジナル) /